Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider being more selective in save() in prepare_distributed in run_future_lapply #436

Closed
kendonB opened this issue Jun 29, 2018 · 3 comments

Comments

@kendonB
Copy link
Contributor

kendonB commented Jun 29, 2018

prepare_distributed contains the following call to save which is quite a bottleneck:

save(
    list = ls(globalenv(), all.names = TRUE),
    envir = globalenv(),
    file = globalenv_file(config$cache_path)
  )

Is it possible to refine this so that it only saves the items that are needed to build the targets to be made? This also seems unnecessary for many types of futures and seems to be replicating a function of the future package itself, which is really good at sending objects around different processes.

@wlandau
Copy link
Member

wlandau commented Jun 29, 2018

We mostly need save() for Makefile parallelism. I think we can get away with just saving the imports. We can probably also suppress it altogether for "future" parallelism.

@wlandau
Copy link
Member

wlandau commented Jun 30, 2018

I would prefer to handle this one after #440, specifically #440 (comment).

@wlandau
Copy link
Member

wlandau commented Jul 4, 2018

f247d27 excludes targets in the plan from being saved. That is the best I can do safely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants