Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get the worker to do the loading in future parallelism? #539

Closed
kendonB opened this issue Oct 9, 2018 · 4 comments
Closed

Get the worker to do the loading in future parallelism? #539

kendonB opened this issue Oct 9, 2018 · 4 comments
Labels
status: priority topic: performance

Comments

@kendonB
Copy link
Contributor

kendonB commented Oct 9, 2018

Giving future parallelism another try. With a lot of targets, all the loading is quite rough on the main process. Is it feasible to structure the dispatch so that the the worker loads dependencies?
@kendonB kendonB changed the title Get the worker to do the loading in future parallelism Get the worker to do the loading in future parallelism? Oct 9, 2018
@wlandau
Copy link
Member

wlandau commented Oct 9, 2018

Sounds reasonable for make(caching = "worker").

@wlandau
Copy link
Member

wlandau commented Oct 10, 2018

I also plan to implement this for `make(parallelism = "clustermq", caching = "worker"). Slightly related: #449 (comment).

@wlandau wlandau mentioned this issue Oct 10, 2018
Closed
@wlandau
Copy link
Member

wlandau commented Oct 10, 2018 •
edited
Loading

Also, I just noticed that "future" parallelism needs checksums to make sure the data writes back correctly before the master process moves on. I will roll in a patch.

wlandau-lilly added a commit that referenced this issue Oct 10, 2018
@wlandau-lilly
Clustermq worker caching means workers also load the dependencies.
18ed599 
ref: #539
@wlandau wlandau mentioned this issue Oct 12, 2018
Merged
6 tasks
@wlandau
Copy link
Member

wlandau commented Oct 12, 2018

Fixed via #544.

@wlandau wlandau closed this as completed Oct 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: priority topic: performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wlandau @kendonB