Try drake + future.batchtools

[wlandau-lilly]

From ropensci/drake#42:

library(future)
library(future.batchtools)
library(drake)

backend(batchtools_slurm) # same as future::plan()

# Alternatively, plan to send targets to SLURM and then give each target multiple cores.
# This should work if you use `future_lapply()` somewhere within a target's command.
# backend(list(batchtools_slurm, multicore))

# Cap the max jobs with options(mc.cores = 2) or something similar from ?future.options
load_basic_example()
make(my_plan, parallelism = "future_lapply")

[wlandau-lilly]

Edit: I am having more trouble than I previously anticipated. Batchtools backends require configuration template files.

backend(batchtools_slurm(template = "batchtools.slurm.tmpl"))

I need to access SLURM and learn batchtools in earnest before I can make this functionality useful. In the meantime, the specially-configured Makefile parallelism in ropensci/drake#115 may help. There is more configurability now than just a shell.sh.