Number of tests (total, failed, etc.) vary wildly

[berndgoetz]

When running our tests, the reported numbers of test executions vary wildly. Here are some outcomes, based on the same code base, server and test client:

Tests run: 2622, Failures: 14, Errors: 0, Skipped: 8
Tests run: 2609, Failures: 4, Errors: 0, Skipped: 1
Tests run: 2302, Failures: 0, Errors: 404, Skipped: 1268
Tests run: 2623, Failures: 0, Errors: 0, Skipped: 0
Tests run: 2302, Failures: 0, Errors: 404, Skipped: 1268
Tests run: 2609, Failures: 4, Errors: 0, Skipped: 5

I haven't figured out the math here. Adding up the numbers don't match the overall number of tests. Any explanations? Any idea how to fix/improve this? Thanks.

[AlexeyBuzdin]

This appears to be a global issue with cucumber cucumber/cucumber-jvm#577 and cucumber/cucumber-jvm#263.
Cucumber report is generated correctly though.
Managed to create a workaround with TeamCity and Bamboo configurations.
Will look into the issue further

[AlexeyBuzdin]

The surefire integration in Cucumber isn't that stable at all, so a possible solution is to create a stand-alone maven/gradle plugin and use cucumber.json as a source for reports. The .json itself is generated correctly and supported in Bamboo.
The issue will be solved with #45