NuGet packaging should support symlinks within packages.

[vdanilin]

Details about Problem

NuGet product used (NuGet.exe | Visual Studio | MSBuild.exe | dotnet.exe): nuget.exe CLI via Mono

Product version: 5.8.0.6930

Worked before? If so, with which NuGet version: No

Operating Systems: Linux and macOS

We create nuget packages on Linux and macOS by ‘nuget.exe pack ..’. Our source folder contains libraries and the set of symbolic links following Linux Shared Library Versioning Guidelines that we expect to be included into nuget package archive as symbolic links. But during packaging, the nuget.exe tool converts the symbolic links into files, which leads to 3x package size increase. As a result, some packages exceed 250Mb limit and cannot be deployed to Nuget gallery. Please fix the nuget.exe tool to support symbolic links.

[nkolev92]

Hey @DanVev,

I was able to repro the behavior on Windows, so it's consistent. As such I've retitle the issue to make it clear it's a feature ask.

While package size is a concern, we'd need to consider how older clients may work with these new packages.

[JonDouglas]

This might be dependent on dotnet/runtime#24271

[nkolev92]

This is not high on our priority list right now.

If we were to ever to this, we'd need to make sure that the sym links don't point outside of the package.

[mattleibow]

Just adding a note here that is is required for mac catalyst frameworks since they are symlinks all over the show. I do have a more specific case with that where I could zip it up and then pack the zip - and then the SDK will unzip when building the app. But, this issue is the big generic version of various tooling that will use symlinks.

The security of the fact with symlinks reaching out is a thing - and might also affect zip files, right?

[lewing]

This is causing pretty severe package size bloat in some workloads where binaries must be copied to maintain the expected behavior.

[carlossanlop]

I work for the dotnet/runtime Libraries team. This bug is preventing me from creating unit test data containing hard links or symbolic links, that needs to get packed in a *.nupkg and then extracted in the unit test execution.

Here's how I repro'd this issue:

• I created some test data in this location: https://github.com/carlossanlop/runtime-assets/tree/tarAssets/src/System.IO.Compression.TestData/TarTestData/gnutarfolders . Notice some of the folders have hardlink or link in their name. If you navigate into them, you'll find actual hardlinks and symbolic links that GitHub is capable of storing accurately.

• All these folders get consumed by runtime/src/libraries/System.IO.Compression/tests/System.IO.Compression.Tests.csproj and get loaded when we restore the test project via this MSBuild property: https://github.com/dotnet/runtime/blob/ed9ab1bd947104c1fc1a2b1c8d80285cc37f863d/eng/Versions.props#L126

• Unfortunately, nuget extracts the contents of the nupkg file and saves the hardlinks and symlinks as normal files (or if it's a directory symlink, it gets saved as a normal directory).

[mattleibow]

I work around this for now by creating a zip of the folder and extracting it on the mac into the obj directory.
https://github.com/mono/SkiaSharp/blob/main/binding/SkiaSharp/nuget/build/maccatalyst/SkiaSharp.targets

[admayber]

Azure internal dev here. +1 that this feature would also help us, particularly with packaging Linux binaries.

[danmoseley]

Note that in .NET 6+ there are new API's to handle symbolic links eg File/Directory.CreateSymbolicLink, and File/Directory.ResolveLinkTarget. Just curious, @carlossanlop do you think that the proposal above would need more than that, eg., FileSystemInfo.IsLink?

[emmenlau]

This issue is also affecting #12136. It makes the Linux shared libraries unusable when the library exists multiple times (instead of symbolic links).

[emmenlau]

I'd like to apply the same workaround as @mattleibow , to package a zip archive with the symbolic links and extract them during NuGet installation or deployment. This was actually quite difficult to get to work. For other people facing the same issue, here is my workaround:

The zip-file is stored in my NuGet as build\native\raw.zip. So there is a corresponding line in my *.nuspec file:

  <files>
    <file src="build\MyProject.targets" target="build" />
    <file src="build\native\raw.zip" target="build\native" />
  </files>

The zipfile should not contain root folders, only the files and folders exactly how they should be extracted to the target directory (typically the OutDir bin of the dependent project). As an example, I place my files in folder build/native/raw, and then zip them with:

cd build/native/raw
zip -0 -q -r "." "../raw.zip"

Then the *.targets file can use a post-build task for the extraction:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <!--
    This is a workaround to store and extract symbolic links on Unix in a zip
    archive. Idea taken from issue https://github.com/NuGet/Home/issues/10734
  -->
  <Target Name="MyProject_NativeLibraryExtraction" AfterTargets="Build" Condition=" '$(OS)' == 'Unix' ">
    <Message Importance="high" Text="MyProject extracting native libraries (including symbolic links) using command: unzip -q -d '$(OutDir)' '$(MSBuildThisFileDirectory)native/raw.zip'" />
    <Exec Command="unzip -q -d '$(OutDir)' '$(MSBuildThisFileDirectory)native/raw.zip'" />
  </Target>
</Project>

[borrrden]

This is causing terrible damage to mac catalyst native frameworks. Here is what I have found:

The structure is as follows:

MyLib -> Symlink to Versions/A/MyLib
Headers -> Symlink to Versions/A/Headers
etc
---- Versions
---- Current -> Symlink to A
---- A
----    ---- MyLib -> actual file
----    ---- Headers -> Actual headers

Without the top level symlink present, the mac catalyst app will not build as it says error MT158: The file '<truncated>/MyLib.framework/MyLib' does not exist

Without the Versions/A/MyLib file present, the app immediately crashes at runtime with library not found

If both are files (copies of each other) the app fails to build claiming the the framework bundle format is ambiguous.

Is there no way to win here? The zipping workaround seems specific to mac but what if this is Windows tethered to mac? Will that still work @emmenlau ?

[mattleibow]

I think the macos and ios sdks now correctly handle zips. No need for any msbuild targets to do this. I think I updated SkiaSharp to just let the sdk do the work.

[emmenlau]

@mattleibow how does that work in detail? You still use the zip-in-nuget workaround, but it extracts automatically?

[borrrden]

@emmenlau I looked at the SkiaSharp project (which is actually pretty nicely structured) to find the answer, and then tried it myself and it looks like the answer to your question is "yes"

[deccer]

Hi guys, I was also looking for hints to solve this symlink problem because of linux' numbering of shared libraries. First I thought adding a symlink to the nuget should work but then I tried this:

Instead of

        <Content Include="../Lib/linux-x64/libktx.so.4.2.1">
            <PackagePath>runtimes/linux-x64/native</PackagePath>
            <Pack>true</Pack>
            <Visible>false</Visible>
        </Content>

I ask PackagePath to simply rename libktx.so.x.x.x to libktx.so. Then NativeLibrary can pick it up on the consuming side.

        <Content Include="../Lib/linux-x64/libktx.so.4.2.1">
            <PackagePath>runtimes/linux-x64/native/libktx.so</PackagePath>
            <Pack>true</Pack>
            <Visible>false</Visible>
        </Content>

[emmenlau]

I ask PackagePath to simply rename libktx.so.x.x.x to libktx.so.

Great if that works for you. However it is sadly not a generic solution to the problem. There are quite many cases where in a single application dependency tree, a library may be sometimes referred to by the name with three digits, with two digits, one digit, or no digit at all. This can happen if a library is more "fundamental" and therefore used by several other libraries, and these other libraries have different version level compatibility requirements.

If, in such cases, you provide only the base name library, dynamic loading may or may not work, and lead to quite weired errors. Furthermore, if you provide all the different namings as copies of the same library, then symbol resolution may be badly broken.

All in all, great if it works for your use case. But it may break at any point in time in the future, when your dependencies change.

[deccer]

Yep, I am aware of that.

[KUGA2]

We use nuget with vcpkg binary cache on linux.
On cached builds, libs are duplicated, on local builds files are symlinks. This caused some confusion.

I did some tests and learned of this issue here. Nugets on the cache-server contain duplicates not links.

When deploying the application to the target, the binaries are also duplicated and cause some bloat.

[m-kuhn]

When caching macos libs this can lead to problems, read crashes:

objc[7050]: Class QMacAutoReleasePoolTracker is implemented in both vcpkg_installed/arm64-osx-dynamic/lib/libQt5Core.5.15.14.dylib (0x105ea85c0) and vcpkg_installed/arm64-osx-dynamic/lib/libQt5Core.5.dylib (0x1133645c0). One of the two will be used. Which one is undefined.