Avoid hanging processes in the lapply-like backends #373
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Previously, there were problems with hanging processes in the
mclapply
andparLapply
backends when one of the processes exited in error. In this PR, I useon.exit()
to enforce the following.Also, the error log of the process is cached in the
"mc_fail"
storr
namespace. For example, retrieve withconfig$cache$get("2", namespace = "mc_fail")
for worker 2 andconfig$cache$get("0", namespace = "mc_fail")
for master. I haven't wrote a friendly API function to retrieve these types of errors, and I currently do not have plans for one. This error caching is mostly just to help me and the contributors debug. Will revisit if these errors come up a lot, but I am expecting most errors to be related to targets once #369 is solved, and there is already error handling for that.Currently affects
mclapply
andparLapply
parallelism, and will affectfuture_lapply
parallelism going forward (hopefully soon).GitHub issues addressed
Checklist
drake
's code of conduct, and I agree to follow its rules.testthat
unit tests totests/testthat
to confirm that any new features or functionality work correctly.devtools::check()