Reduced .NET8 R2R performance

[JensNordenbro]

Description

On arm64, using PublishReadyToRun on the attached project; you will get reduced performance using .NET8 (not present in NET7, NET6).

dotnet publish .\Arm64Net8JitError.csproj -f net6.0-windows -r win-arm64 -c Release --no-self-contained -p:PublishReadyToRun=true -o error_62_net6
 
dotnet publish .\Arm64Net8JitError.csproj -f net8.0-windows -r win-arm64 -c Release --no-self-contained -p:PublishReadyToRun=true -o error_63_net8

On my machine:
NET6: 10ms
NET8: 650ms

To help you guys get started, we have extracted a minimal reproduce project that triggers the error from a real application if you wonder why the app has no apparent purpose.

Changing from struct to class on Point2D will remove the problem.
Arm64Net8JitError.zip

Reproduction Steps

Start program published with parameter PublishReadyToRun:true and press the one button and get a messagebox with the init time to spot the difference.

Expected behavior

Non regressed performance.

Actual behavior

Really bad performance -> real world applicaton starts 10 times slower. Attached benchmark app is even slower than that.

Regression?

Yes, works in NET6.0, NET7.0

Known Workarounds

Convert Point2D struct to class instead.

Configuration

See description.

Other information

We target imx8MPlus.

[janvorli]

cc: @trylek

[JensNordenbro]

Any comments on this issue?

[steveisok]

@ivdiazsa please run this through the ringer and see if you get the same results.

[AndyAyersMS]

Looks like in .NET 8 there is one method with unusually long jitting time (found via the jit stats tab of a perfview run) that likely accounts for the startup time issue you're seeing:

that method does not appear in the .NET 6 jit stats, so presumably it was prejitted in .NET 6 but not in .NET 8.

So the question is, why doesn't that method get prejitted in .NET 8? That will take a bit more digging.

[AndyAyersMS]

Seems like per r2r dump the method does get prejitted, so perhaps the runtime is deciding the method can't be used for some reason? (runtime might also toss the entire R2R payload).

There is an ETW bit in the method load event for this, but perfview doesn't parse it yet in the raw payload.

[ivdiazsa]

Thank you very much for the insight Andy. What might be the possible reasons the runtime might decide that and discard everything? And how would we investigate such a case?

[AndyAyersMS]

In order to use R2R code, the runtime performs a number of checks to decide if the prejitted image (as a whole) and then each individual method within that image can be safely used in the new run.

Looks like setting DOTNET_ReadyToRunLogFile=some-filename will give you some logging info, but from what I see it's not detailed enough to figure out what is going on

Ready to Run initialized successfully: "C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.6\System.Private.CoreLib.dll".
Ready to Run initialized successfully: "C:\bugs\r96242\Arm64Net8JitError\error_63_net8\Arm64Net8JitError.dll".
Ready to Run header not found: "C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.6\System.Runtime.dll".
...

@VSadov @jkotas can you share the right diagnostic steps to try at this point?

[jkotas]

So the question is, why doesn't that method get prejitted in .NET 8?

Pass --singlemethodtypename Arm64Net8JitError.Shapes_Polyline1,Arm64Net8JitError --singlemethodname InitializeComponent --verbose switch to crossgen2.exe to diagnose this. The verbose output will have the error that prevented the method from getting prejitted:

Processing 1 dependencies
Compiling [Arm64Net8JitError]Arm64Net8JitError.Shapes_Polyline1.InitializeComponent()
Warning: Method `[Arm64Net8JitError]Arm64Net8JitError.Shapes_Polyline1.InitializeComponent()` was not compiled because: Failed to load type 'Arm64Net8JitError.PointD' from assembly 'Arm64Net8JitError, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'

Then run crossgen2 under debugger to figure our what went wrong. You should see an exception thrown at this callstack:

 # Child-SP          RetAddr               Call Site
00 00000047`ccbfddb8 00007ff6`dfafbdf7     crossgen2!RhpThrowEx [D:\a\_work\1\s\src\coreclr\nativeaot\Runtime\amd64\ExceptionHandling.asm @ 111] 
01 00000047`ccbfddc0 00007ff6`dfa73df8     crossgen2!ILCompiler_TypeSystem_Internal_TypeSystem_ThrowHelper__ThrowTypeLoadException_0+0x37 [/_/src/coreclr/tools/Common/TypeSystem/Common/ThrowHelper.cs @ 17] 
02 00000047`ccbfde10 00007ff6`dfa7343f     crossgen2!ILCompiler_ReadyToRun_ILCompiler_CompilerTypeSystemContext__EnsureLoadableTypeUncached+0x498 [/_/src/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.Validation.cs @ 233] 
03 00000047`ccbfde80 00007ff6`dfa73a0e     crossgen2!ILCompiler_ReadyToRun_ILCompiler_CompilerTypeSystemContext__EnsureLoadableType+0x6f [/_/src/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.Validation.cs @ 82] 
04 00000047`ccbfded0 00007ff6`dfa7343f     crossgen2!ILCompiler_ReadyToRun_ILCompiler_CompilerTypeSystemContext__EnsureLoadableTypeUncached+0xae [/_/src/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.Validation.cs @ 252] 

@ivdiazsa You should be able to take it from here.

[jkotas]

@ivdiazsa I see that you have modified the milestone to 10.0. The PointD type is very simple. It looks like a pretty bad crossgen2 bug to me if we are not able to load it. You may want to reconsider.

[JensNordenbro]

@ivdiazsa, for us, labling this 10.0, makes the prospect of moving from net6.0-lts problematic. Either we stay=) insecure or we move to net8.0 =) slow.

[steveisok]

@ivdiazsa, for us, labling this 10.0, makes the prospect of moving from net6.0-lts problematic. Either we stay=) insecure or we move to net8.0 =) slow.

This is something we'll prioritize now and is highly likely to make its way back to net8.

[ivdiazsa]

Thanks a lot for the detailed explanation and guidance @jkotas. Will prioritize it and get to investigate it on Monday.

[ivdiazsa]

@JensNordenbro Hi, we've been researching and are currently developing a fix. While investigating, we found that your repro app has an [assembly: Guid("Guid goes here")] annotation in Properties/AssemblyInfo.cs file. Considering you also have [assembly: ComVisible(false)], is there a reason you might need the Guid? I ask because due to where the bug lies in the type system, if you remove that Guid clause, the regression disappears. I got ~5ms on my testing machine, which is fairly similar to .NET 6 and .NET 7. Would this work as a temporary workaround?

[JensNordenbro]

Hi @ivdiazsa !

I will try to schedule a real investigation on the real app but I think it might take some time before we get to that. (summer, ...)

• I will get back with our results unless you guys beet me to the punch!

Nice to see progress!

[ivdiazsa]

Hi @JensNordenbro! We've finished working on the fix and will soon begin working on the backport to .NET 8.

[ivdiazsa]

Hi @JensNordenbro! We've finished working on the fix and will soon begin working on the backport to .NET 8.

The backport has been successfully completed. The fix will be available on the .NET 8.0.10 release.

[JensNordenbro]

Thanks @ivdiazsa!
Once started, this was quick 😄