Add an option to return an error code if the tests failed

[mattleibow]

I have to add custom code to detect any failed tests in the results so I can fail the build.

If there was a command line arg to basically use the test failure to also mean the command failed. This can wait until everything is cleaned up, but should still return a non 0 exit code if my tests failed.

[mattleibow]

Maybe this is a "bug" since iOS actually fails on test failures...

[premun]

As far as I know, XHarness exits with 1 if tests fail (at least on iOS):

xharness/src/Microsoft.DotNet.XHarness.Common/CLI/ExitCode.cs

Line 21 in 66b9a52

TESTS_FAILED = 1,

Or is this about Android only?

[mattleibow]

This is Android only.

[premun]

I see that the way it is currently handled in dotnet/runtime is that the wrapper itself returns 1 when it detects a failed test in the TestRunner:
https://github.com/dotnet/runtime/blob/372248b9b4e3e7c08e0c0aa53d4a94f826fe273b/src/libraries/Common/tests/AndroidTestRunner/AndroidTestRunner.cs#L34-L38

In XHarness we currently do this:

• iOS - we cannot reliable detect exit code and we understand the contents of testResults.xml enough to tell and then exit with 1
• Android - we can detect exit code of the Android run easily and just return that. However, we don't even open testResults.xml and we don't try to guess from that.

Question 1: Is it a lot of trouble for you to do what runtime does?
Question 2: Should we unify this across XHarness and have Android analyze test results? (prolongs run time, is more complicated)

[mattleibow]

That runner appears to be a console app? Is it not possible for xharness to look at the exit cote returned by the bundle and determine a fail?

info: Running instrumentation class com.mono.skiasharp.tests.TestInstrumentation took 123.3714867 seconds
dbug: Exit code: 0
      Standard Output:
      INSTRUMENTATION_RESULT: return-code=1
      INSTRUMENTATION_RESULT: test-execution-summary=Tests run: 4651 Passed: 4595 Inconclusive: 0 Failed: 1 Ignored: 55
      INSTRUMENTATION_RESULT: test-results-path=/storage/emulated/0/Documents/TestResults.xml
      INSTRUMENTATION_CODE: -1
      
      
info: Found XML result file: '/storage/emulated/0/Documents/TestResults.xml'(key: test-results-path)
info: Attempting to pull contents of /storage/emulated/0/Documents/TestResults.xml to /Users/runner/work/1/s/output/testlogs/SkiaSharp.Android.Tests
dbug: Executing command: '/Users/runner/.dotnet/tools/.store/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21116.1/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21116.1/runtimes/any/native/adb/macos/adb -s emulator-5554 pull /storage/emulated/0/Documents/TestResults.xml /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cewsmccz.odp'
dbug: Copied 1 files to /Users/runner/work/1/s/output/testlogs/SkiaSharp.Android.Tests.
info: Test execution summary:
      Tests run: 4651 Passed: 4595 Inconclusive: 0 Failed: 1 Ignored: 55
info: Instrumentation finished normally with exit code 1
dbug: Executing command: '/Users/runner/.dotnet/tools/.store/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21116.1/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21116.1/runtimes/any/native/adb/macos/adb -s emulator-5554 logcat -d '
info: Wrote current ADB log to /Users/runner/work/1/s/output/testlogs/SkiaSharp.Android.Tests/adb-logcat-com.mono.skiasharp.tests.log
fail: Non-success instrumentation exit code: 1, expected: 0
info: Attempting to remove apk 'com.mono.skiasharp.tests': 
dbug: Executing command: '/Users/runner/.dotnet/tools/.store/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21116.1/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21116.1/runtimes/any/native/adb/macos/adb -s emulator-5554 uninstall com.mono.skiasharp.tests'
info: Successfully uninstalled com.mono.skiasharp.tests.
XHarness exit code: 0

There is also the case of a test crash:

dbug: Exit code: 0
      Standard Output:
      INSTRUMENTATION_RESULT: shortMsg=Process crashed.
      INSTRUMENTATION_CODE: 0
      
      
info: Short Message: 
      Process crashed.
fail: No value for 'return-code' provided in instrumentation result.  This may indicate a crashed test (see log)
dbug: Executing command: '/Users/runner/.dotnet/tools/.store/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21111.1/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21111.1/runtimes/any/native/adb/macos/adb -s emulator-5554 logcat -d '
info: Wrote current ADB log to /Users/runner/work/1/s/output/testlogs/SkiaSharp.Android.Tests/adb-logcat-com.mono.skiasharp.tests.log
dbug: Executing command: '/Users/runner/.dotnet/tools/.store/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21111.1/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21111.1/runtimes/any/native/adb/macos/adb -s emulator-5554 bugreport /Users/runner/work/1/s/output/testlogs/SkiaSharp.Android.Tests/adb-bugreport-com.mono.skiasharp.tests.zip'
info: Wrote ADB bugreport to /Users/runner/work/1/s/output/testlogs/SkiaSharp.Android.Tests/adb-bugreport-com.mono.skiasharp.tests.zip
fail: Non-success instrumentation exit code: 82, expected: 0
info: Attempting to remove apk 'com.mono.skiasharp.tests': 
dbug: Executing command: '/Users/runner/.dotnet/tools/.store/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21111.1/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21111.1/runtimes/any/native/adb/macos/adb -s emulator-5554 uninstall com.mono.skiasharp.tests'
info: Successfully uninstalled com.mono.skiasharp.tests.
XHarness exit code: 0

I see fail: Non-success instrumentation exit code: 82, expected: 0, but then the xharness app is all good?

It seems xharness itself succeeds even though we just asked it to fail? At least that is the disconnect for me.

[premun]

Oh, I didn't know about this. This looks like a bug. @greenEkatherine can you please have a look what we're doing with the app's exit code and the expected exit code?

[greenEkatherine]

Yeah, it should be fixed. What we do now - we compare adb process' exit code with the expected one. But this adb exit code doesn't reflect what happens with instrumentation