If test fails in hooks, send test 'name' as original test to reporters instead of including 'before each hook for....' text in test 'name'

[juan-fernandez]

Current behavior:

When you're using the default JUnit reporter, as shown in https://on.cypress.io/reporters#Command-line-1:

cypress run --reporter junit --reporter-options "mochaFile=results/my-test-output.xml,toConsole=true"

With a test similar to:

describe('test', () => {
  beforeEach(() => {
    cy.get('#hola')
  })
  it('can do stuff', () => {
    cy.get('#stuff')
       .should('exist')
  })
})

The xml output is this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="5.875" tests="1" failures="0">
  <testsuite name="Root Suite" timestamp="2020-03-05T17:22:24" tests="0" failures="0" time="0">
  </testsuite>
  <testsuite name="test" timestamp="2020-03-05T17:22:24" tests="1" failures="0" time="5.875">
    <testcase name="test can do stuff" time="5.875" classname="can do stuff">
    </testcase>
  </testsuite>
</testsuites>

So far so good.

The problem starts when an error happens in beforeEach (e.g., #element is no longer on the screen). Then the result is the following:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0" tests="1" failures="1">
  <testsuite name="Root Suite" timestamp="2020-03-05T17:24:18" tests="0" failures="0" time="0">
  </testsuite>
  <testsuite name="test" timestamp="2020-03-05T17:24:18" tests="1" failures="1" time="0">
    <testcase name="test &quot;before each&quot; hook for &quot;can do stuff&quot;" time="0" classname="&quot;before each&quot; hook for &quot;can do stuff&quot;">
      <failure message="Timed out retrying: Expected to find element: &apos;#hola&apos;, but never found it.

Because this error occurred during a &apos;before each&apos; hook we are skipping the remaining tests in the current suite: &apos;test&apos;" type="CypressError"><![CDATA[CypressError: Timed out retrying: Expected to find element: '#hola', but never found it.

Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite: 'test'
    at Object.cypressErr (https://shared.scope.dev/__cypress/runner/cypress_runner.js:105262:11)
    at Object.throwErr (https://shared.scope.dev/__cypress/runner/cypress_runner.js:105217:18)
    at Object.throwErrByPath (https://shared.scope.dev/__cypress/runner/cypress_runner.js:105249:17)
    at retry (https://shared.scope.dev/__cypress/runner/cypress_runner.js:96526:16)
    at onFailFn (https://shared.scope.dev/__cypress/runner/cypress_runner.js:85591:16)
    at tryCatcher (https://shared.scope.dev/__cypress/runner/cypress_runner.js:139407:23)
    at Promise._settlePromiseFromHandler (https://shared.scope.dev/__cypress/runner/cypress_runner.js:137343:31)
    at Promise._settlePromise (https://shared.scope.dev/__cypress/runner/cypress_runner.js:137400:18)
    at Promise._settlePromise0 (https://shared.scope.dev/__cypress/runner/cypress_runner.js:137445:10)
    at Promise._settlePromises (https://shared.scope.dev/__cypress/runner/cypress_runner.js:137520:18)
    at Async../node_modules/bluebird/js/release/async.js.Async._drainQueue (https://shared.scope.dev/__cypress/runner/cypress_runner.js:134132:16)
    at Async../node_modules/bluebird/js/release/async.js.Async._drainQueues (https://shared.scope.dev/__cypress/runner/cypress_runner.js:134142:10)
    at Async.drainQueues (https://shared.scope.dev/__cypress/runner/cypress_runner.js:134016:14)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

The problem is that name within testcase changes to "test "before each" hook for "can do stuff"" because of the error in beforeEach, even though the test is exactly the same as before.

This makes it impossible for tools that ingest this kind of reports to unequivocally identify tests.

Desired behavior:

The name in the testcase should be constant whether the test fails or not.

Test code to reproduce

describe('test', () => {
  beforeEach(() => {
    cy.get('#hola')
  })
  it('can do stuff', () => {
    cy.get('#stuff')
       .should('exist')
  })
})

with failing beforeEach run with:

cypress run --reporter junit --reporter-options "mochaFile=results/my-test-output.xml,toConsole=true"

Versions

Cypress package version: 3.7.0
Cypress binary version: 3.7.0

Running on macOS 10.15.3

[joselee]

Any updates with this? I'm facing the exact issue. Failures in hooks are manipulating the name and className, and this is messing things up for reporting in CI.

[dkuebric]

Also running into this issue with internal test monitoring tooling parsing cypress test output.

[jennifer-shehane]

I don't see us changing the name of the test in this case - this is what we use to display in the Command Log and stdout - in order to be more descriptive about where the error is throwing in order to more quickly debug.

Perhaps there is some way to change the name that is sent directly to reporters for use. I'm not sure.

[midleman]

Any updates on this? Also running into this issue. The only workaround at the moment is to avoid using before or beforeEach hooks, but that's not a solution. :(

[Ghostbird]

I'm running into this as well. My dashboard is full of "tests" with a zero percent pass percentage.

[nicoarg]

Other tests frameworks (like pytest) fail with the test name if a setup/teardown failed. This is easy then to post to an external management tool that automatically maps your jUnit file with tests so I agree on only the tests names must be used for these cases. I am coding a workaround to edit the jUnit file before being posted but would be nice to have this issue resolved directly in Cypress or as an option.

[shashankgumte]

I am also facing same issue and due to that I am not able to update the generated .xml files in to testrail.
Any fix for this?

[Roemer]

We are also facing this issue. Any solutions or workarounds?

Edit1:
There are various other places where this issue is mentioned.
michaelleeallen/mocha-junit-reporter#181
bhovhannes/junit-report-merger#127
which ultimatively lead to a bug in xmlbuilder which was fixed a few weeks ago here: oozcitak/xmlbuilder2#154
So maybe updating to that version of xmlbuilder2 in Cypress could finally solve the issue?

Edit2:
I poked around more. In a mocha reporter, the name in the EVENT_TEST_FAIL (test.title) is mangled but when looping thru the suites and tests in the EVENT_RUN_END, all titles are actually correct.

Edit3:
I get the intention, that not the test failed but the "beforeEach", so the code before the test. But as mentioned before, usually the "beforeEach" code belongs to each test as well and should therefore in fact fail the test instead of a thing in-between tests. So I would appreciate a change to handle it that way (like pretty much all other testing frameworks).

[emilyrohrbough]

I just ran a simple test to verify how mocha handles reporting this (i.e. didn't test using Cypress) and the reporting is the same.

Command:

mocha --reporter mocha-junit-reporter 'scripts/unit/example-spec.js'

Spec with Before Each error:

describe('example, () => {
  beforeEach(() => {
    expect(true).to.be.false
  })

  it('should pass', () => {
    expect(true).to.be.true
  })

Generated Report:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.0000" tests="7" failures="1">
  <testsuite name="Root Suite" timestamp="2023-04-07T17:58:58" tests="0" time="0.0000" failures="0">
  </testsuite>
  <testsuite name="example" timestamp="2023-04-07T17:58:58" tests="7" file="<>/scripts/unit/example-spec.js" time="0.0000" failures="1">
    <testcase name="example &quot;before each&quot; hook for &quot;should pass&quot;" time="0.0000" classname="&quot;before each&quot; hook for &quot;should pass&quot;">
      <failure message="expect is not defined" type="ReferenceError"><![CDATA[ReferenceError: expect is not defined
    at Context.<anonymous> (scripts/unit/example-spec.js:6:5)
    at processImmediate (node:internal/timers:466:21)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

This is not a Cypress-specific issue since this is how mocha is reporting this information.

[damhonglinh]

Switching to cypress-circleci-reporter (from mocha-junit-reporter) seems to solve this issue for me.

cypress-circleci-reporter seems to also report file attribute, and use the original test name (rather than "before each ...") in classname