Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preprocess up-to-date targets in mc_master #455

Closed
kendonB opened this issue Jul 4, 2018 · 4 comments
Closed

Preprocess up-to-date targets in mc_master #455

kendonB opened this issue Jul 4, 2018 · 4 comments

Comments

@kendonB
Copy link
Contributor

kendonB commented Jul 4, 2018

I believe this will go a long way to solving the issue I was having in #449.

You probably could literally call outdated in this function, being careful to ensure outdated uses jobs_imports wherever it lightly parallelizes.

@wlandau
Copy link
Member

wlandau commented Jul 4, 2018

It gets tricky because sometimes mc_master() is used to process the imports at the same time. But maybe we can remove up-to-date targets from the schedule here. That way, they will never get added to the priority queue.

@wlandau wlandau closed this as completed in 95d3f95 Jul 4, 2018
@kendonB
Copy link
Contributor Author

kendonB commented Oct 9, 2018

@wlandau If all targets are up to date, make_targets still runs run_parallel_backend unnecessarily. Since the worker launches but ends up doing nothing, this can still take a long time if the HPC scheduler doesn't launch your job right away. Would checking the length (however you do that for an igraph) of config$schedule do it?

@wlandau
Copy link
Member

wlandau commented Oct 9, 2018

Sure. For distributed backends, it does make sense to check the schedule in make_targets().

@wlandau
Copy link
Member

wlandau commented Oct 9, 2018

Fixed in de44794.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants