enhance PyTorch easyblock to print individual failed tests #2983
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(created using
eb --new-pr
)This enhances the output in the log file by printing not only the test suite of failed tests but also the individual failed tests. This should make it easier to determine if there is only a single test in a test suite that fails for everyone (or most) so we can disable that single test
Edit: As identified in easybuilders/easybuild-easyconfigs#18421 (comment) there are more issues with those possible solutions:
run_test.py
to not shard at all (any maybe disable the exit-on-first-failure)export BUILD_ENVIRONMENT=slow-gradcheck
as a hack to disable the parallelization: https://github.com/pytorch/pytorch/blob/v1.13.1/test/run_test.py#L721 although that might get forgotten when the code changes (and it seems to have changed in PyTorch 2 already and was only introduced in 1.13)--save-xml
) which might be the best option but requires quite some work.Edit: As for 4.:
It needs 2 Python packages (i.e. builddependencies):
lxml
andunittest-xml-reporting
and a patch for PyTorch to propagate--save-xml
.But then it has folders named after the tests with 1 or more xml files containing e.g.
<testsuites><testsuite name="pytest" errors="0" failures="0" skipped="127" tests="476" time="408.891" timestamp="2023-08-16T13:49:53.750990" hostname="taurusi8002"><testcase classname="TestJitCPU" name="test_jit_alias_remapping_abs_cpu_float32" time="0.063" file="test_ops_jit.py" />
which does look helpful