Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Improve Cypress test stability for Longview, Linode Rescue, Linode Rebuild flows #9902

Merged
merged 21 commits into from
Nov 29, 2023

Conversation

jdamore-linode
Copy link
Contributor

@jdamore-linode jdamore-linode commented Nov 14, 2023

Description 📝

Attempts to improve stability for some of our flakier tests.

Will circle back with a self review and better explanation for some of these changes soon. Hoping for extra scrutiny for the changes in src

Changes 🔄

  • Refactors Longview end-to-end test and related Longview installation script to greatly reduce Longview failures and improve performance
  • Reduces flakiness of Linode rebuild and rescue tests by waiting for form data to load before attempting to submit (more context here)
  • Fixes a logic issue in chooseRegions() helper that occasionally causes the same region to appear multiple times in the returned array -- this would occasionally manifest as test failures in the Linode firewall migration test
  • Fixes a test flake issue in VPC assign and unassign tests by making component selection more specific
  • Re-enables Cypress video recording, which is disabled by default in Cypress 13

How to test 🧪

We can lean on CI for this, but I'll update this PR with a command that runs only the modified tests for local testing.

As an Author I have considered 🤔

Check all that apply

  • 👀 Doing a self review
  • ❔ Our contribution guidelines
  • 🤏 Splitting feature into small PRs
  • ➕ Adding a changeset
  • 🧪 Providing/Improving test coverage
  • 🔐 Removing all sensitive information from the code and PR description
  • 🚩 Using a feature flag to protect the release
  • 👣 Providing comprehensive reproduction steps
  • 📑 Providing or updating our documentation
  • 🕛 Scheduling a pair reviewing session
  • 📱 Providing mobile support
  • ♿ Providing accessibility support

@jdamore-linode jdamore-linode self-assigned this Nov 14, 2023
@jdamore-linode jdamore-linode requested a review from a team as a code owner November 14, 2023 17:58
@jdamore-linode jdamore-linode requested review from bnussman-akamai and abailly-akamai and removed request for a team November 14, 2023 17:58
Copy link
Contributor

@abailly-akamai abailly-akamai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor looks good! Tested a few locally a it all passed ✅

Are we worried longview.spec.ts failed in the CI here? Locally passed but seems to have gone over the time limit in the CI?

return new Array(count).fill(null).reduce((acc: Region[], _cur, index) => {
const chosenRegion: Region = ((): Region => {
if (index === 0 && overrideRegion) {
return overrideRegion;
}
// Get an array of regions that have not already been selected.
const unusedRegions = regions.filter(
(regionA: Region) =>
!!regions.find((regionB: Region) => regionA.id !== regionB.id)
(region: Region) => !acc.includes(region)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make more sense to do an includes on the regionId rather than the whole object?

Copy link
Contributor Author

@jdamore-linode jdamore-linode Nov 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll double check this but I think this should be ok -- the comparison happens by reference (i.e. includes() returns true if passed a reference to an object in the array, but returns false if passed a different but identical object). It's not actually doing a value comparison if e.g. performance is a concern, and it should be sound since all of the objects we're working with come from the same regions array that we're comparing against.

@jdamore-linode
Copy link
Contributor Author

Refactor looks good! Tested a few locally a it all passed ✅

Are we worried longview.spec.ts failed in the CI here? Locally passed but seems to have gone over the time limit in the CI?

Thanks @abailly-akamai! The Longview test is failing in CI because the Docker container doesn't have expect installed -- this never manifested as an error before because we've been suppressing the errors. I've pushed a couple commits that I hope will fix that now, but I do see some VPC test failures that I need to figure out too.

@@ -327,6 +331,7 @@ export const RebuildFromStackScript = (props: Props) => {
<ActionsPanel
primaryButtonProps={{
'data-testid': 'rebuild',
'data-qa-form-data-loading': isLoading,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I understand what's going on here, but it feels weird to me to not have a user-facing loading state and to have data-qa-form-data-loading specifically for testing.

I don't think this should hold anything up... Just wondering if there is a more elegant way to go about it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I thought about this and sort of arrived here as a compromise, even though it's kind of a hack:

  • I wanted to avoid changing anything user-facing since this PR is already pretty broad in scope, and I think making a UI change here could warrant a little more discussion (see below)
  • I figure we can strip this out pretty easily if we do implement a UI improvement later on, such that the test can rely on the state of the UI rather than a data attribute

Adding a loader was my initial thought too, but thinking about it more eventually had me questioning whether it would actually be a better UX.

There's only a very brief period before the data gets populated, and I was concerned that a loader would essentially result in a quick flash that wouldn't appear long enough to convey to the user what is happening, if that makes sense. Meanwhile, it happens so quickly that I don't think a real user can interact with the form fast enough to trigger the issue if they tried (even Cypress is only fast enough to trigger the issue a small percent of the time).

So ultimately I just had doubts about what the best solution was, but if we have consensus that a loader (or some other solution) is the way to go, I'd be glad to take care of it here! But if not I'm hoping this solution is okayish enough to hold us over until we come up with something better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This solution is definitely okay! All of that is fair!

@jdamore-linode jdamore-linode added Approved Multiple approvals and ready to merge! and removed Ready for Review labels Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Approved Multiple approvals and ready to merge!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants