Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EMSDK-.WasmNativeDefaultsTests work item failing #67887

Closed
runfoapp bot opened this issue Apr 12, 2022 · 9 comments · Fixed by #69519
Closed

EMSDK-.WasmNativeDefaultsTests work item failing #67887

runfoapp bot opened this issue Apr 12, 2022 · 9 comments · Fixed by #69519
Labels
arch-wasm WebAssembly architecture area-VM-meta-mono

Comments

@runfoapp
Copy link

runfoapp bot commented Apr 12, 2022

Runfo Tracking Issue: EMSDK-.WasmNativeDefaultsTests work item failing

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1758177 runtime PR 68991 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1756138 runtime PR 68126 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1754800 runtime PR 66147 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log runclient.py
1754259 runtime PR 66147 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log runclient.py
1752402 runtime PR 66147 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log runclient.py
1750315 runtime Rolling net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1749104 runtime PR 68801 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1744053 runtime PR 66304 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1729244 runtime PR 67883 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1725717 runtime PR 68077 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1725600 runtime Rolling net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1718025 runtime PR 68024 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1716274 runtime Rolling net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1712733 runtime PR 67773 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1712633 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1712034 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1711717 runtime PR 67676 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1711187 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1710393 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1710148 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1709042 runtime PR 61776 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1708510 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
0 0 8
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@radical
Copy link
Member

radical commented Apr 12, 2022

Issue is being tracked at dotnet/arcade#9025

@radical radical added the arch-wasm WebAssembly architecture label Apr 12, 2022
@ghost
Copy link

ghost commented Apr 12, 2022

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo Tracking Issue: EMSDK-.WasmNativeDefaultsTests work item failing

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1712733 runtime PR 67773 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1712633 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1712034 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1711717 runtime PR 67676 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log core dump test results runclient.py
1711187 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1710393 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1710148 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1709042 runtime PR 61776 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py
1708510 runtime PR 67771 net7.0-Browser-Release-wasm-Mono_Release_Linux-BuildWasmApps-Ubuntu.1804.Amd64.Open console.log test results runclient.py

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
4 9 9
Author: runfoapp[bot]
Assignees: -
Labels:

arch-wasm, untriaged, area-VM-meta-mono

Milestone: -

@radical radical removed the untriaged New issue has not been triaged by the area owner label Apr 12, 2022
@jakobbotsch
Copy link
Member

That issue was closed @radical.
There are now new failures in the latest rolling build that look related to the Emscripten SDK:

[upv2xixw.rdh]   non_async_main_args_Release_False -> /datadisks/disk1/work/AA0C094A/w/9FA308C4/e/upv2xixw.rdh/bin/Release/net7.0/browser-wasm/publish/
[upv2xixw.rdh]   Traceback (most recent call last):
[upv2xixw.rdh]     File "/datadisks/disk1/work/AA0C094A/p/build/emsdk/upstream/emscripten/emcc.py", line 23, in <module>
[upv2xixw.rdh]       from tools.toolchain_profiler import ToolchainProfiler
[upv2xixw.rdh]   ModuleNotFoundError: No module named 'tools'
[upv2xixw.rdh] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error : Traceback (most recent call last): [/datadisks/disk1/work/AA0C094A/w/9FA308C4/e/upv2xixw.rdh/non_async_main_args_Release_False.csproj]
[upv2xixw.rdh] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error :   File "/datadisks/disk1/work/AA0C094A/p/build/emsdk/upstream/emscripten/emcc.py", line 23, in <module> [/datadisks/disk1/work/AA0C094A/w/9FA308C4/e/upv2xixw.rdh/non_async_main_args_Release_False.csproj]
[upv2xixw.rdh] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error :     from tools.toolchain_profiler import ToolchainProfiler [/datadisks/disk1/work/AA0C094A/w/9FA308C4/e/upv2xixw.rdh/non_async_main_args_Release_False.csproj]
[upv2xixw.rdh] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error : ModuleNotFoundError: No module named 'tools' [/datadisks/disk1/work/AA0C094A/w/9FA308C4/e/upv2xixw.rdh/non_async_main_args_Release_False.csproj]
[upv2xixw.rdh] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error MSB3073: The command "emcc --version" exited with code 1. [/datadisks/disk1/work/AA0C094A/w/9FA308C4/e/upv2xixw.rdh/non_async_main_args_Release_False.csproj]

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1750315&view=results
Example log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-5d82cf334bf34431b6/EMSDK-.MainWithArgsTests/1/console.acdd695a.log?helixlogtype=result

@jakobbotsch jakobbotsch added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 4, 2022
@jakobbotsch
Copy link
Member

jakobbotsch commented May 5, 2022

A new EMSDK failure that failed the latest rolling build, this time in FlagsChangeRebuildTest:

Binlog path: /datadisks/disk1/work/A95A0970/w/A3FD0907/uploads/xharness-output/logs/kt1z0fri.c1b/rebuild_flags_Release.binlog
Building rebuild_flags_Release in /datadisks/disk1/work/A95A0970/w/A3FD0907/e/kt1z0fri.c1b
Running: /datadisks/disk1/work/A95A0970/p/sdk-no-workload/dotnet: publish -p:WasmBuildOnlyAfterPublish=true  /p:WasmBuildSupportDir=/datadisks/disk1/work/A95A0970/p/build /p:EMSDK_PATH=/datadisks/disk1/work/A95A0970/p/build/emsdk  /p:Configuration=Release /bl:"/datadisks/disk1/work/A95A0970/w/A3FD0907/uploads/xharness-output/logs/kt1z0fri.c1b/rebuild_flags_Release.binlog" /nologo /fl /flp:"v:diag,LogFile=/datadisks/disk1/work/A95A0970/w/A3FD0907/uploads/xharness-output/logs/kt1z0fri.c1b/rebuild_flags_Release.binlog.log" /v:minimal  
WorkingDirectory: /datadisks/disk1/work/A95A0970/w/A3FD0907/e/kt1z0fri.c1b
    Wasm.Build.NativeRebuild.Tests.FlagsChangeRebuildTest.ExtraEmccFlagsSetButNoRealChange(buildArgs: BuildArgs { ProjectName = placeholder, Config = Release, AOT = False, ProjectFileContents = placeholder, ExtraBuildArgs =  }, extraCFlags: "/p:EmccExtraCFlags=-g", extraLDFlags: "/p:EmccExtraLDFlags=-g", host: V8, id: "utva41aq.bgq") [FAIL]
      Build process exited with non-zero exit code: 139
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/tests/BuildWasmApps/Wasm.Build.Tests/BuildTestBase.cs(589,0): at Wasm.Build.Tests.BuildTestBase.AssertBuild(String args, String label, Boolean expectSuccess, IDictionary`2 envVars, Nullable`1 timeoutMs)
        /_/src/tests/BuildWasmApps/Wasm.Build.Tests/BuildTestBase.cs(338,0): at Wasm.Build.Tests.BuildTestBase.BuildProject(BuildArgs buildArgs, String id, BuildProjectOptions options)
        /_/src/tests/BuildWasmApps/Wasm.Build.Tests/NativeRebuildTests/NativeRebuildTestsBase.cs(50,0): at Wasm.Build.NativeRebuild.Tests.NativeRebuildTestsBase.FirstNativeBuild(String programText, Boolean nativeRelink, Boolean invariant, BuildArgs buildArgs, String id, String extraProperties)
        /_/src/tests/BuildWasmApps/Wasm.Build.Tests/NativeRebuildTests/FlagsChangeRebuildTest.cs(36,0): at Wasm.Build.NativeRebuild.Tests.FlagsChangeRebuildTest.ExtraEmccFlagsSetButNoRealChange(BuildArgs buildArgs, String extraCFlags, String extraLDFlags, RunHost host, String id)
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.RuntimeMethodInfo.InvokeNonEmitUnsafe(Object obj, IntPtr* arguments, Span`1 argsForTemporaryMonoSupport, BindingFlags invokeAttr)

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1750315&view=results
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-2043f5bb2a2e432d80/EMSDK-.FlagsChangeRebuildTest/1/console.601b2900.log?helixlogtype=result
There is a coredump available.

@lewing
Copy link
Member

lewing commented May 5, 2022

that failure looks like dotnet/msbuild#7596

@jakobbotsch jakobbotsch removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 5, 2022
@jakobbotsch
Copy link
Member

The build error further up looks like emscripten-core/emsdk#130, which indicates the installation of the emscripten SDK somehow was not successful. Let's see if it comes back.

@jakobbotsch
Copy link
Member

After looking further back, I found the same failure in https://dev.azure.com/dnceng/public/_build/results?buildId=1750315&view=ms.vss-test-web.build-test-results-tab&runId=47225478&resultId=100050&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab:

[qpalugss.kk5]   Traceback (most recent call last):
[qpalugss.kk5]     File "/datadisks/disk1/work/AA0C094A/p/build/emsdk/upstream/emscripten/emcc.py", line 23, in <module>
[qpalugss.kk5]       from tools.toolchain_profiler import ToolchainProfiler
[qpalugss.kk5]   ModuleNotFoundError: No module named 'tools'
[qpalugss.kk5] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error : Traceback (most recent call last): [/datadisks/disk1/work/AA0C094A/w/AE130951/e/qpalugss.kk5/invariant_unset.csproj]
[qpalugss.kk5] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error :   File "/datadisks/disk1/work/AA0C094A/p/build/emsdk/upstream/emscripten/emcc.py", line 23, in <module> [/datadisks/disk1/work/AA0C094A/w/AE130951/e/qpalugss.kk5/invariant_unset.csproj]
[qpalugss.kk5] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error :     from tools.toolchain_profiler import ToolchainProfiler [/datadisks/disk1/work/AA0C094A/w/AE130951/e/qpalugss.kk5/invariant_unset.csproj]
[qpalugss.kk5] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error : ModuleNotFoundError: No module named 'tools' [/datadisks/disk1/work/AA0C094A/w/AE130951/e/qpalugss.kk5/invariant_unset.csproj]
[qpalugss.kk5] /datadisks/disk1/work/AA0C094A/p/build/wasm/WasmApp.Native.targets(474,5): error MSB3073: The command "emcc --version" exited with code 1. [/datadisks/disk1/work/AA0C094A/w/AE130951/e/qpalugss.kk5/invariant_unset.csproj]

That means the error is persistent. Can we review the emscripten SDK installation process on the CI machines and consider if we can add some logic to validate the installation and reinstall it when this happens?

@jakobbotsch jakobbotsch added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' and removed blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' labels May 6, 2022
@jakobbotsch
Copy link
Member

Sorry, I seem to have mixed my browser tabs up, those are the same failures, so let's keep a watch out to see if it is persistent...

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label May 20, 2022
radical added a commit that referenced this issue May 20, 2022
- Run all the debugger tests in one helix job, because most of them take a couple of minutes to run. And this would avoid the overhead of individual helix jobs

- Add a no-workloads variant for `Wasm.Build.Tests`, instead of `EMSDK`
  - This replaces the job that used `WasmApp.LocalBuild*` stuff to build
with a `EMSDK` checkout. That case already gets tested with library tests.
   - The `EMSDK` case had been disabled recently due to an Arcade issue
  - Since we have very few tests to run for the noworkloads case, they are
all run in a single helix work item.

Fixes #58159
Fixes #67887 (by completely removing the `EMSDK` run)
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label May 20, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jun 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-wasm WebAssembly architecture area-VM-meta-mono
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants