Improve our use of automated accessibility tools

[36degrees]

What

Extend our use of Axe (and/or other automated testing tools) to cover more of our examples, and run in an environment where JavaScript and CSS are included and run.

Why

#1966 was caught downstream by a user who was running Axe against their application. Improving the way we use automated testing tools will improve the chances of issues like this being caught before they can be released.

Further detail

At the minute, we use jest-axe on some (but not all) of the examples for each component, but it runs against the markup only – it’s not run in an environment where CSS / JS are included. We may want to remove these tests if we are able to test in an environment where JS / CSS is included.

We do have other tests that run in Puppeteer for components that use JavaScript, but we do not use Axe in Puppeteer.

Done when…

• we've identified which examples or pages are beneficial to test
• tests are run against those examples
• tests are run in an environment where CSS and JavaScript are included
• any redundant tests are removed
• issues identified in the tests are reported clearly on the command line
• the build fails when issues are identified
• we have a way to silence false positives (for example, issues that are raised as a result of testing a component in isolation)

[36degrees]

From talking to a developer at DfE where issue #1966 was caught, this is their setup:

This is our end to end testing repository: https://github.com/DFE-Digital/apply-for-teacher-training-tests/

It's a git repo that contains a cypress end to end test, nothing fancy or unusual. The tests are triggered via github actions whenever we deploy qa/staging/production

We're using cypress-axe in a couple of ways:

• We have a test that runs as a "candidate" user, so it signs in and tries to fulfill the main journey. When the user hits the home page, we run a function we defined call .runAxe, which injects the Axe javascript into the page and then checks the accessibility, facilitated by the cypress-axe plugin

• Separately from checking the homepage we also have "component preview pages" (think Design System kitchen sink / a component on its own inside of a web page, with predefined arguments). We also walk through those in a separate test and run Axe against all of them. That tests the building blocks we use to build the rest of the pages

https://dashboard.cypress.io/projects/tqfe7m/runs/1009/failures is an example of the failing test that caught the issue in #1966.

They suggested we could look at using axe-puppeteer to do something similar.

[36degrees]

This also came up as part of a 'tech & ops health check' exercise the team ran in November 2020, as part of the 'testing – pre-deployment' practise.

We scored ourselves as 'good' for that practise, but said that we were 'moderately concerned' about it as we know there are some areas that could be improved. This was flagged as one of the things that we wanted to do.

[colinrotherham]

We ended up closing this (and including all component examples) in:

• Switch Accessibility tests to @axe-core/puppeteer #3522