MSBuild NuGet Content Asset handling alternately creates and deletes content every other build

[idg10]

SDK version: .NET Core SDK 3.1.200 (and also 3.1.102)

When a .NET project has a reference to a NuGet package that includes a content asset to be compiled into a project (e.g., the Nullable NuGet package adds a Nullable.cs source file to the compilation for projects for some target frameworks), if you run builds twice in succession, they alternate between copying the file into the obj folder and deleting it.

This causes a problem for other parts of the build that want to use the file if they happen to run at the wrong time. For example, the problem described at coverlet-coverage/coverlet#714 (comment) appears to arise from this.

In that case, Coverlet wants to be able to read the source file as part of its code coverage handling. But because of this issue, alternate builds can end up failing to produce any code coverage information, because on every other build, the Nullable.cs file is missing.

The cause appears to be the behaviour of the IncrementalClean target (in Microsoft.Common.CurrentVersion.targets). This seems to be trying to remove any intermediate build outputs that were produced by a previous build but which it believes are no longer required. Unfortunately, in cases where the content asset in question was put in place by an earlier build, the RunProduceContentAssets target (in Microsoft.PackageDependencyResolution.targets) does not list the file in its FileWrites output (because the file was already there so it didn't write it), with the effect that the IncrementalClean has no way of knowing that this content asset is still wanted.

The result is that IncrementalClean removes a file because it mistakenly thinks it is a hangover from an older build that is no longer required.

This often goes unnoticed because the relevant cleanup happens after compilation, so it typically doesn't greatly matter if the file vanishes. But when parts of the build want to inspect source code later on (as Coverlet does—it wants to inspect the source as part of its instrumentation process) this overly aggressive cleanup causes problems.

(This is causing some of our CI builds to fail to produce code coverage information, because we run the test with a separate dotnet test step after the dotnet build, and since that dotnet test is the 2nd build to occur, it falls foul of this problem. And as far as we can tell, the obvious workaround of specifying --no-build won't work because the coverage process relies on instrumentation that is performed as part of the build.)

[kevinoid]

I'm experiencing the same issue with Nullable and coverlet. I also wanted to note that the symptoms (and cause via FileWrites) are similar to item 3 in #1898. Perhaps they can be addressed together?

P.S. Thanks @idg10! It would have taken me ages to figure this all out without your detailed investigation.

[yuehuang010]

+1, hitting this exact issue.

[jimmylewis]

@dsplaisted I noticed in the PR you mentioned that this was only fixed in 7.0. What would be the right process to request this being ported to 6.0? Our incremental builds are broken 50% of the time with this issue.

[dsplaisted]

@jimmylewis I've submitted #23656 to backport the fix for 6.0.3xx, which will correspond to VS 17.2.

We would need to go through more of an approval process if we want to put it in 6.0.2xx or 6.0.1xx servicing.