[RNMobile][Monorepo] Use github actions to run React Native e2e tests #22189
Conversation
|
Size Change: 0 B Total Size: 828 kB ℹ️ View Unchanged
|
ceyhun
force-pushed
the
try/github-actions-ios-ci
branch
from
8f36267 to
94a1e41
Compare
ceyhun
changed the title
ceyhun
changed the title
ceyhun
requested review from
Tug and
gziolo
and removed request for
karmatosed,
youknowriad and
mapk
| @@ -60,6 +60,10 @@ describe( 'Gutenberg Editor tests for Block insertion', () => { | |||
| await paragraphBlockElement.click(); | |||
|
|
|||
| await editorPage.addNewParagraphBlock(); | |||
|
|
|||
| // wait for accessibility ids to update | |||
| await driver.sleep( 1000 ); | |||
There was a problem hiding this comment.
Oh nice that's why I had some failures locally and on travis I think 👍
| npm ci | ||
| npm run native test:e2e:android:local | ||
|
|
||
| - uses: actions/upload-artifact@v2 |
There was a problem hiding this comment.
Could you explain a bit how this works? Where are the artifacts stored? For how long? How can one access it when looking at a PR?
This could be a nice addition to the mobile doc, otherwise just a comment on this PR so people can find this information using git blame :)
There was a problem hiding this comment.
Nevermind, found them here at the top right of the check screen
There was a problem hiding this comment.
In case it's useful, there is some more information here about GitHub actions artifacts: https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts.
@ceyhun 👋 😄 nice work on this! I ❤️ seeing these workflows integrated with GitHub!
Tug
added
the
Mobile App - i.e. Android or iOS
|
Excellent idea to use GitHub actions and great work integration all that into branch 👍 How long does it take to run those tests?
47 minutes? Travis takes 20 minutes at most to run all jobs, an individual job finishes in maximum of 13-14 minutes. Do you have some ideas on how to make it faster? Do you know if there are any limits to GH actions? |
ceyhun
force-pushed
the
try/github-actions-ios-ci
branch
from
55ed0d6 to
08c90a9
Compare
|
@gziolo GH actions have a concurrency limit of 5 for macOS jobs and 20 in total: https://help.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits. Currently both Android and iOS are running on macOS. We have some ideas but we'd like to iterate on this later on. For now we thought we'll just run a subset of the tests in parallel to make it faster and so this PR can be merged. It should be around ~15min now. |
I knew there needs to be some sort of limit. I'm wondering whether there is some OSS plan that we could request. I guess it would be already enabled. There is also an option to use self-hosted jobs. 2 existing actions use Ubuntu, now we would have 4 more actions running macOS, so it still looks good.
It's fine for now, but it'd be great to optimize it as much as possible – ideally aiming at 5-10 minutes per job. (Not sure if that is realistic though, boot process for WordPress env takes nearly 5 minutes on Travis ...) |
That would be awesome indeed but that might be quite hard to achieve, I believe it takes around 10mins at least to just "build" the apps. The upside of parallelisation is that we can keep adding e2e tests while staying around 15mins total.
Given the recent contributions to |
…squash' into try/github-actions-ios-ci
There was a problem hiding this comment.
I don't know much of anything about the native testing, and only a fairly basic comprehension of GitHub Actions, so I'm maybe not the best person to review 😄 But from what I can grasp, it appears to be in good shape.
Like @gziolo , I'm especially conscious of the timing of these checks. Unfortunately the existing testing isn't especially reliable, so it's quite common to need to restart them from time to time. I guess if these are outside of Travis, any such rebuilds wouldn't need to run this process again, which is good at least.
| @@ -60,6 +60,10 @@ describe( 'Gutenberg Editor tests for Block insertion', () => { | |||
| await paragraphBlockElement.click(); | |||
|
|
|||
| await editorPage.addNewParagraphBlock(); | |||
|
|
|||
| // wait for accessibility ids to update | |||
| await driver.sleep( 1000 ); | |||
There was a problem hiding this comment.
I'm not sure what other options might exist in the native testing, but in web code it's typically discouraged to use arbitrary timeouts, due to (a) slowing down test runtime and (b) not guaranteeing to always work (example). In Puppeteer tests, we'd instead actually express this as whatever it is that's being waited for (e.g. waitForSelector instead of waitFor( timeout )). Are there any equivalent options here?
There was a problem hiding this comment.
We're using xpath to find our elements on native so we could indeed use driver.waitForElementByXPath to speed up tests on faster devices.
Tbh we have quite many anti-patterns on those native e2e tests that we're hoping to fix. This PR was supposed to only be a simple port from gutenberg-mobile so we were not supposed to touch this code but since we did I guess we could improve the code a little bit there 👍
There was a problem hiding this comment.
This should be gone now, updated the tests with the ones from gutenberg-mobile/develop. setImplicitWaitTimeout was also initially added there and I had increased it.
| @@ -41,6 +41,8 @@ export default class EditorPage { | |||
| this.accessibilityIdKey = 'contentDescription'; | |||
| this.accessibilityIdSuffix = ', '; | |||
| } | |||
|
|
|||
| driver.setImplicitWaitTimeout( 10000 ); | |||
There was a problem hiding this comment.
I see 🤔 By default this is 0, so setting this to 10s, makes sure each and every element query is "transformed" into its equivalent waitFor... neat!
| @@ -114,8 +114,8 @@ | |||
| "test:e2e:bundle:android": "mkdir -p android/app/src/main/assets && react-native bundle --reset-cache --platform android --dev false --minify false --entry-file index.js --bundle-output android/app/src/main/assets/index.android.bundle --assets-dest android/app/src/main/res", | |||
| "test:e2e:build-app:android": "npm run test:e2e:bundle:android && cd android && ./gradlew clean && ./gradlew assembleDebug", | |||
| "test:e2e:install-app:android": "cd android && ./gradlew installDebug", | |||
| "test:e2e:bundle:ios": "react-native bundle --reset-cache --platform=ios --dev=false --minify false --entry-file=index.js --bundle-output=./ios/build/gutenberg/Build/Products/Release-iphonesimulator/GutenbergDemo.app/main.jsbundle --assets-dest=./ios/build/gutenberg/Build/Products/Release-iphonesimulator/GutenbergDemo.app", | |||
| "test:e2e:build-app:ios": "npm run ios --configuration Release --no-packager", | |||
| "test:e2e:bundle:ios": "mkdir -p ios/build/gutenberg/Build/Products/Release-iphonesimulator/GutenbergDemo.app && react-native bundle --reset-cache --platform=ios --dev=false --minify false --entry-file=index.js --bundle-output=./ios/build/gutenberg/Build/Products/Release-iphonesimulator/GutenbergDemo.app/main.jsbundle --assets-dest=./ios/build/gutenberg/Build/Products/Release-iphonesimulator/GutenbergDemo.app", | |||
There was a problem hiding this comment.
Could we call this script from test:e2e:ios:local? Otherwise we have to call it manually each time
There was a problem hiding this comment.
Bundling should automatically happen when test:e2e:build-app:ios is called via a script in the iOS Xcode project. In the CI case we call them seperately for caching purposes and add SKIP_BUNDLING=true flag before calling test:e2e:build-app:ios
There was a problem hiding this comment.
Agreed for automated CI using github actions. But for manually running e2e tests locally (which is what test:e2e:ios:local is used for atm) we need to automatically build the js bundle. I'll add it
There was a problem hiding this comment.
I wanted to say it was already happening during test:e2e:build-app:ios locally. There's this script in Xcode:
https://github.com/facebook/react-native/blob/0.61-stable/scripts/react-native-xcode.sh
This is invoked by Xcode via react-native run-ios when test:e2e:build-app:ios is called and should create the bundle already.
But now it returns an error saying gutenberg/index.js was not found. Maybe that's why test:e2e:ios:local was returning an error locally .
If we create the bundle before react-native-xcode.sh is invoked, it exits without throwing an error
So it's fine, for now 🙂
|
Per @Tug 's ping over chat, will now merge this PR with the "Commit merge" strategy and lift the setting right afterwards. |
Description
This is part of a migration effort of gutenberg-mobile to gutenberg repository with git history. More related PRs here: #18508
This PR aims to get all mobile e2e tests working on the most up to date gb-mobile / gutenberg Monorepo branch (feat/import-gutenberg-mobile-no-squash) using GitHub actions
Bonus: The emulator screens are recorded while the tests are running and uploaded as artifacts to help debugging.
How has this been tested?
The following two commands should work, with all tests passing:
npm run native test:e2e:android:localnpm run native test:e2e:ios:localPR check on this PR should be green for the tasks:
React Native E2E Tests (Android)React Native E2E Tests (iOS)Screenshots
Types of changes
Checklist: