on('fail') not reliably continuing test after failure

[sebastianclavijo]

Current behavior

The cy.on('fail') code below should catch the 2 failed assertions (and print in the console, and and finalize the test as passed.
However the test fails/stops in the first failed assertion, and even do not enter in the cy.on('fail') function:

/// <reference types="Cypress" />
describe('suite', () => {
    it('test', () => {
        cy.on('fail', (err,runnable) => {
            console.log("*****Error")
            console.log(err)
            return false
        })
        cy.visit('https://google.com')
        cy.get('[data-cy]') // Assertion fails (but should continue testing)
        cy.wrap(false).should('be.false') // Assertion pass
        cy.wrap(false).should('be.true') // Assertion fails (but should continue testing)
    });
});

BTW: I also tried Cypress.on('fail') within the describe outside de test and in the e2e.js with no success.

Desired behavior

No response

Test code to reproduce

/// <reference types="Cypress" />
describe('suite', () => {
    it('test', () => {
        cy.on('fail', (err,runnable) => {
            console.log("*****Error")
            console.log(err)
            return false
        })
        cy.visit('https://google.com')
        cy.get('[data-cy]') // Assertion fails (but should continue testing)
        cy.wrap(false).should('be.false') // Assertion pass
        cy.wrap(false).should('be.true') // Assertion fails (but should continue testing)
    });
});

Cypress Version

13.4.0

Node version

18.14.2.

Operating System

Windows 10

Debug Logs

No response

Other

No response

[csvan]

"Title" should be a short text describing the issue, not your professional title.

[jennifer-shehane]

Yah I'm seeing some different, but similar behavior when I run this.

This code does not run the cy.wrap commands. I feel like this may be expected, but maybe poorly documented in our docs.

/// <reference types="Cypress" />
describe('suite', () => {
    it('test', () => {
        cy.on('fail', (err,runnable) => {
            console.log("*****Error")
            console.log(err)
            return false
        })
        cy.visit('https://google.com')
        cy.get('[data-cy]') // Assertion fails (but should continue testing)
        cy.wrap(false).should('be.false') // Does not run at all
        cy.wrap(false).should('be.true') 
    });
});

[MikeMcC399]

According to the comments in

https://github.com/cypress-io/cypress-example-recipes/blob/master/examples/fundamentals__errors/cypress/e2e/test-fails.cy.js

this would be expected:

    cy.visit('index.html')
    cy.get('button#does-not-exist', { timeout: 1000 }).click()

    // note: after the cy.get fails and the test fails
    // the remaining commands are NOT executed
    // thus this failing assertion never gets to run
    cy.wrap(false).should('be.true')

(Issue is also posted to https://discord.com/channels/755913899261296641/1168655650994270339/1168655650994270339.)

[jennifer-shehane]

Opened a PR to docs to clarify this: cypress-io/cypress-documentation#5554

[sebastianclavijo]

There is a pretty good article https://www.lambdatest.com/learning-hub/exception-handling-in-cypress#fail describing the expected result when catching an error with cy.on('fail').
As you can see in the screenshots of the article, the tests finish gracefully as not failed in green (what is in red is only the assertion that failed).

However in the screenshots of my sample, the test fails (in red), and even more in the screenshot you can see in the console that it never enters in the handler function associated to the cy.on('fail'). So independently if the rest of assertions are not run, the event is not properly handled (even more, if the code is aborted, what's the point of capturing the 'fail' event?).

Is it missing some configuration option somewhere, to get the 'fail' event captured?