RestoreTaskEx needs to use a response file when calling NuGet.Build.Tasks.Console.exe

[jaredpar]

NuGet Product Used

MSBuild.exe

Product Version

Visual Studio 17.7 but guessing any existing release has this behavior

Worked before?

The bug has seemingly always existed but was recently exposed due to our increase in code size

Impact

I'm unable to use this version

Repro Steps & Context

Repro steps:

1. Clone https://github.com/dotnet/roslyn
2. Run Build.cmd -restore -pack

This will error with

...\NuGet.RestoreEx.targets(19,5): error : The filename or extension is too
long [Z:\src\Features\LanguageServer\Microsoft.CodeAnalysis.LanguageServer\Microsoft.CodeAnalysis.LanguageServer.csproj]

The reason is RestoreTaskEx project shells out to NuGet.Build.Tasks.Console.exe. One of the arguments it passes is a mapping of projects in the restore plus, their absolute path and several configuration.

Example:

  <ProjectConfiguration Project="{4B45CA0C-03A0-400F-B454-3D4BCB16AF38}" AbsolutePath="C:\Users\jaredpar\code\roslyn\src\Compilers\CSharp\csc\AnyCpu\csc.csproj" BuildProjectInSolution="True">Debug|AnyCPU</ProjectConfiguration>

I'm unsure what criteria is used here (assume it's just transitive dependencies) but it does scale with the number of included projects. That means once enough projects are included in the restore operation the command argument can exceed the maximum command line length. This will result in the RestoreTaskEx task failing as it exceeds this limit.

Believe that RestoreTaskEx and NuGet.Build.Tasks.Console.exe should be communicating via a response file. This is common practice when shelling out from a build task to a tool that takes command lines whose length scales with the size of the build input.

Verbose Logs

Can provide a binary log if this helps with the diagnosis. Here is the full invocation for Nuget.Build.Tasks.Console.exe

https://gist.github.com/jaredpar/7d73ac0106613d6b0c9a375043d3b9f6

[jaredpar]

Small update. The restore operation which triggers this error is coming from a nested <MSBuild> invocation. One of our team members discovered we can work around this by disabling static graph on that restore operation

        <MSBuild Projects="$(MSBuildProjectFullPath)" Targets="Restore" Properties="RestoreWithR2R=true;RestoreUseStaticGraphEvaluation=false" />

That gives us a temporary work around. Hopefully helps pinpoint the issue a bit for you all :)

[jeffkl]

Duplicate of #11968

[jeffkl]

I tried fixing this once and it caused a regression so we reverted it since not very many people hit the original issue. NuGet/NuGet.Client#4772

[kartheekp-ms]

Duplicate of # 11968 see above #12843 (comment)

[jaredpar]

Will move conversation over to #11968. It's a bit discouraging to see that particular issue has been opened for over a year now with no resolution. This is a frustratingly hard problem to diagnose when hit and the build is unusable until you root cause it.

[jeffkl]

Will move conversation over to #11968. It's a bit discouraging to see that particular issue has been opened for over a year now with no resolution. This is a frustratingly hard problem to diagnose when hit and the build is unusable until you root cause it.

@jaredpar in NuGet/NuGet.Client#5413 it will now automatically detect if the command-line argument string is too long and serialize global properties over STDIN. I also added better error handling which will log much clearer errors if something happens during argument parsing.