Skip to content

Commit

Permalink
Fix #369
Browse files Browse the repository at this point in the history
  • Loading branch information
wlandau-lilly committed May 6, 2018
1 parent 095c4c0 commit ea17878
Show file tree
Hide file tree
Showing 8 changed files with 21 additions and 193 deletions.
9 changes: 4 additions & 5 deletions R/config.R
Original file line number Diff line number Diff line change
Expand Up @@ -262,12 +262,11 @@
#' `make(..., parallelism = "future_lapply", lazy_load = "bind")`.
#'
#' If `lazy_load` is `"eager"`,
#' drake prunes the execution environment before every
#' parallelizable stages, removing all superfluous targets
#' and then loading any dependencies it will need
#' for the targets in the current parallelizable stage.
#' drake prunes the execution environment before each target/stage,
#' removing all superfluous targets
#' and then loading any dependencies it will need for building.
#' In other words, drake prepares the environment in advance
#' for all the whole collection of targets in the stage.
#' and tries to be memory efficient.
#' If `lazy_load` is `"bind"` or `"promise"`, drake assigns
#' promises to load any dependencies at the last minute.
#' Lazy loading may be more memory efficient in some use cases, but
Expand Down
9 changes: 1 addition & 8 deletions R/dataframes_graph.R
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,7 @@
#' whether to skip the imports and only include the
#' targets in the workflow plan.
#'
#' @param split_columns logical, whether to break up the
#' columns of nodes to make the aspect ratio of the rendered
#' graph closer to 1:1. This improves the viewing experience,
#' but the columns no longer strictly represent parallelizable
#' stages of build items. (Although the targets/imports
#' in each column are still conditionally independent,
#' there may be more conditional independence than the graph
#' indicates.)
#' @param split_columns logical, deprecated.
#'
#' @param font_size numeric, font size of the node labels in the graph
#'
Expand Down
79 changes: 3 additions & 76 deletions R/parallel_ui.R
Original file line number Diff line number Diff line change
Expand Up @@ -57,82 +57,9 @@ drake_batchtools_tmpl_file <- function(
#' @return Character vector listing the types of parallel
#' computing supported.
#'
#' @details Run `make(..., parallelism = x, jobs = n)` for any of
#' the following values of `x` to distribute targets over parallel
#' units of execution.
#' \describe{
#' \item{'parLapply'}{launches multiple processes in a single R session
#' using \code{parallel::\link{parLapply}()}.
#' This is single-node, (potentially) multicore computing.
#' It requires more overhead than the `'mclapply'` option,
#' but it works on Windows. If `jobs` is `1` in
#' [make()], then no 'cluster' is created and
#' no parallelism is used.}
#'
#' \item{'mclapply'}{uses multiple processes in a single R session.
#' This is single-node, (potentially) multicore computing.
#' Does not work on Windows for `jobs > 1`
#' because [mclapply()] is based on forking.}
#'
#' \item{'future_lapply'}{
#' opens up a whole trove of parallel backends
#' powered by the `future` and `future.batchtools`
#' packages. First, set the parallel backend globally using
#' `future::plan()`.
#' Then, apply the backend to your drake_plan
#' using `make(..., parallelism = "future_lapply", jobs = ...)`.
#' But be warned: the environment for each target needs to be set up
#' from scratch, so this backend type is higher overhead than either
#' `mclapply` or `parLapply`.
#' Also, the `jobs` argument only applies to the imports.
#' To set the max number of jobs, set the `workers`
#' argument where it exists. For example, call
#' `future::plan(multisession(workers = 4))`,
#' then call \code{\link{make}(your_plan, parallelism = "future_lapply")}.
#' You might also try options(mc.cores = jobs),
#' or see future::.options
#' for environment variables that set the max number of jobs.
#' }
#'
#' \item{'future'}{
#' Just like `"future_lapply"` parallelism, except that
#' 1. Rather than use staged parallelism, each target begins as soon as
#' their dependencies are ready and a worker is available. This
#' greatly increases parallel efficiency.
#' 2. The `jobs` argument to `make()` works as intended: in
#' `make(jobs = 4, parallelism = "future")`, 4 workers will run
#' simultaneously, so at most 4 jobs will run simultaneously.
#' 3. The `"future"` backend is experimental, so it is more likely
#' to have bugs than its counterparts.
#' }
#'
#' \item{'Makefile'}{uses multiple R sessions
#' by creating and running a Makefile.
#' For distributed computing on a cluster or supercomputer,
#' try \code{\link{make}(..., parallelism = 'Makefile',
#' prepend = 'SHELL=./shell.sh')}.
#' You need an auxiliary `shell.sh` file for this,
#' and [shell_file()]
#' writes an example.
#'
#' Here, Makefile-level parallelism is only used for
#' targets in your workflow plan
#' data frame, not imports. To process imported objects and files,
#' drake selects the best parallel
#' backend for your system and uses
#' the number of jobs you give to the `jobs`
#' argument to [make()].
#' To use at most 2 jobs for imports and at most 4 jobs
#' for targets, run
#' `make(..., parallelism = 'Makefile', jobs = 2, args = '--jobs=4')`
#'
#' Caution: the Makefile generated by
#' \code{\link{make}(..., parallelism = 'Makefile')}
#' is NOT standalone. DO NOT run it outside of
#' [make()] or [make()].
#' Also, Windows users will need to download and install Rtools.
#' }
#' }
#' @details See the
#' [parallel computing guide](https://ropensci.github.io/drake/articles/parallelism.html) # nolint
#' for details on the parallel backends.
#'
#' @param distributed_only logical, whether to return only
#' the distributed backend types, such as `Makefile` and
Expand Down
9 changes: 1 addition & 8 deletions man/dataframes_graph.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 4 additions & 5 deletions man/drake_config.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 4 additions & 5 deletions man/make.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

81 changes: 3 additions & 78 deletions man/parallelism_choices.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 1 addition & 8 deletions man/vis_drake_graph.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit ea17878

Please sign in to comment.