Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Suite Cleanup #6445

Open
1 of 9 tasks
rhuanjl opened this issue May 9, 2020 · 1 comment
Open
1 of 9 tasks

Test Suite Cleanup #6445

rhuanjl opened this issue May 9, 2020 · 1 comment

Comments

@rhuanjl
Copy link
Collaborator

rhuanjl commented May 9, 2020

There are various issues with the test suite that make it harder to maintain and use than it should be. Each of these should be addressed. However this obviously won't improve the experience for any users of ChakraCore it's a purely for developer/contributor experience hence this is low priority but still something to work on:

  • Tests can't be run on release builds Running tests on release build #6422
  • If a test file does nothing OR does not exist it is counted as a pass CI cleanup #6596
  • To run tests: Linux & macOS use runtests.py, windows uses rl.exe - ideally should be consistent Explore deleting rl.exe #6585 CI cleanup #6596
  • Many tests have unnecessary baseline files when they could just throw exceptions for failures and print 'pass' for success - baselines are strictly necessary only when tracing actions that can't be synchronously monitored/checked within javascript
  • There are 48 tests that don't use a baseline but print a multi-line output ('pass'/'passed' repeatedly)
  • Success indicators vary "Pass", "Passed", "pass", "PASSED" etc. - should be consistent.
  • Many tests are not well explained/don't clearly show what they're for
  • Many test files could be combined into larger aggregate files - which would run faster and be easier to manage
  • Baseline files all use CRLF line endings - ch outputs CRLF on windows but LF only on linux/macOS - currently runtests.py converts CRLF in the baselines to LF only upon every run - should update ch to do LF only and remove all CRs

#6444 makes some progress on some of these points though there is far more to be done.

@rhuanjl
Copy link
Collaborator Author

rhuanjl commented May 12, 2020

Another nice to have, if we drop rl.exe would be to replace the xml files that list all the tests with json, I think it could be a lot cleaner and drastically reduce the boilerplate they contain - but this is certainly not something to change anytime soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant