SIGABRT on Apple M1 (rosetta)

[haraldsteinlechner]

Description

I'm not sure about the "should be" status of dotnet on new apple silicon. anyways, the problem I stumbled across might help in stabilising the support. The problem looks very similar to bullet point two in this issue: #44897

On Mac OS, with x64 emulation I get the following output on console:
RestoreState: 1335: thread_set_state(float) (os/kern) invalid argument

when looking at the dumps it looks as such: https://pastebin.com/tEj95DD4
the relevant part would be:

Thread 1 Crashed:
0   ???                           	0x00007ffe95cce9bc ???
1   libsystem_kernel.dylib        	0x00007fff20390502 __pthread_kill + 10
2   libsystem_c.dylib             	0x00007fff20311720 abort + 120
3   libcoreclr.dylib              	0x0000000108cc1b99 MachExceptionInfo::RestoreState(unsigned int) + 297
4   libcoreclr.dylib              	0x0000000108cc141b SEHExceptionThread(void*) + 571
5   libsystem_pthread.dylib       	0x00007fff203be950 _pthread_start + 224
6   libsystem_pthread.dylib       	0x00007fff203ba47b thread_start + 15

I'm pretty sure the problem is in the runtime for those reasons:

• it does not happen on other architectures
• when digging deeper it seems the problem is very similar to: Support .NET on Apple Silicon with Rosetta 2 emulation #44897 (known issue 2)
• the precondition here:

  runtime/src/coreclr/src/pal/src/exception/machexception.cpp

  Line 984 in fd094a9

  This means: no printf, no TRACE, no PAL allocation, no ExitProcess,

  captured my attention in combination with the following statement from the previous issue: "Rosetta 2 emulation crashes with a fatal failure when calling with thread_get_state x86_FLOAT_STATE64. This is because the emulator does not emulate AVX support, but the function should simply return an error."

unfortunately I have no easy to share minimal repro.

Configuration

• 5.0.100
• Mac mini M1
• System Version: macOS 11.0 (20A2411)
• Kernel Version: Darwin 20.1.0

Regression?

unknown

Other information

related issues:

• Support .NET on Apple Silicon with Rosetta 2 emulation #44897
• GC suspension & UNIX signals corrupt each other's memory on the stack on macOS #44498

if there is an easy fix I'm happy to try asap using my test setup.

thanks and cheers,
Harald

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

@sdmaclea

[danmoseley]

Oops, I read ARM64 not AMD64

[sdmaclea]

/cc @janvorli Yes this looks like the known issue where Rosetta2 doesn't gracefully fall back AVX during emulation

[sdmaclea]

if there is an easy fix I'm happy to try asap using my test setup.

@haraldsteinlechner Testing/building is not trivial, but if you want to try to build and try our current patches the process would look like this.

• Download and decompress .NET 5.0 runtime tar.gz for macos-x64 (path referenced as <net5 root>) using https://dotnet.microsoft.com/download/dotnet-core/scripts/v1/dotnet-install.sh
• checkout https://github.com/dotnet/runtime/tree/release/5.0 (path referenced as <git runtime root>)
• cherry-pick [release/5.0] Use minimum supported PAGE_SIZE as stack probe step #45226
• cherry-pick https://github.com/janvorli/runtime/tree/macos-inject-activation-via-signal-older janvorli@aee81ac
• build src/coreclr/build-runtime.sh -x64 -release
• copy <git runtime root>/artifacts/bin/coreclr/OSX.x64.Release/*.dylib into <net5 root>/shared/Microsoft.NETCore.App/5.0.0

[haraldsteinlechner]

thank you for the detailed explanation. the fixes at least for me look promising :) In fact I tried to naively hack out some portions suspect to failure on rosetta but even failed on compiling the x64 runtime on m1 (I'm not sure how naive this is ;)) - most likely due to missing knowledge of mac tooling as such.

you have a different/real setup for cross-compiling right? As I have seen in the other issue, most tests now pass with #45226 Seems like this one was not necessary:janvorli@aee81ac . In that case my test still could be useful. If by any chance there is a build available I would of course come back. the cross compilation seems to hot for now :(

[janvorli]

@haraldsteinlechner My changes for activation injection via signal is still being tested and there are some issues we are hitting with it. Without this change, you can observe hangs in rare cases when GC needs to suspend a thread. If you don't experience hangs, you don't need to use that one in your experiments.

[sdmaclea]

Two tests deadlocked w/o janvorli@aee81ac
With the change two different tests asserted. Apple is looking at fixing both issues for us see #44897 discussion.

different/real setup for cross-compiling right?

We don't have any cross compile support in the 5.0 branch. The master (6.0) branch can be cross compiled on either macos platform for the other. The trick is to use xcode 12.2 or greater. And add the architecture and /p:CrossBuild=1 to the command line.

[janvorli]

/p:CrossBuild=1

--cross option works too.

[haraldsteinlechner]

now with 11.2 out I checked again and the problem changed - I have another segfault now:

Thread 16 Crashed:
0   ???                           	000000000000000000 0 + 0
1   libsystem_kernel.dylib        	0x00007fff20315f86 swtch_pri + 10
2   libcoreclr.dylib              	0x000000010904f185 PAL_DispatchException + 405
3   libcoreclr.dylib              	0x000000010904ec97 PAL_DispatchExceptionWrapper + 10
4   ???                           	000000000000000000 0 + 0

on: System Version: macOS 11.2 (20D53)
and dotnet --version 5.0.102 (EDIT: also happens with `dotnet --version 5.0.200-preview.21073.2``

is this one familiar or known? @janvorli

EDIT: the full crash dump is here: https://pastebin.com/nxrKLT2h

[janvorli]

@haraldsteinlechner I might have seen this in the past, but there is too little evidence. Does it crash like this consistently? And does it print anything to the console besides crashing? I've done quite a lot of testing on the macOS 11.2 beta 2 and the only issue I've hit was a very rare assert in Rosetta 2: "assertion failed: GPR thread_set_state is unsupported while in sa_tramp. (ThreadContextRegisterState.cpp:1250 thread_set_state_gpr_64)".

[haraldsteinlechner]

yes it is 100% reproduceable, it just crashes without console output. there is little chance i can trim it down do a small example. how could i help in finding the problem?

[janvorli]

From the log you have provided, I can see that your application was running under the .NET Core 3.1.11 runtime:

/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.11/libcoreclr.dylib

Is that expected?

[haraldsteinlechner]

I managed to switch to net5 app. Unfortunately same behaviour: https://pastebin.com/zR0nvQRp
no output except for segmentation fault...

[haraldsteinlechner]

ok.. after digging deeper I found a segfault in our code (in the OpenGL implementation we hit an unimplemented feature). Still a mystery though why it looked like the crash dump previously posted.

EDIT: to make this clear I cannot confirm Dotnet related crashes for now! Thanks for the good work also in the other issues. Will ping back.

[janvorli]

@haraldsteinlechner thank you for the details!

[haraldsteinlechner]

after heavy testing we could not spot any problems - for me this could be closed. @krauthaufen