Add support for automatic disabling of multi-threading while running …

…multicore futures [#355]

DESCRIPTION

@@ -1,5 +1,5 @@
 Package: future
-Version: 1.15.1-9001
+Version: 1.15.1-9005
 Title: Unified Parallel and Distributed Processing in R for Everyone
 Imports:
     digest,
@@ -8,6 +8,7 @@ Imports:
     parallel,
     utils
 Suggests:
+    RhpcBLASctl,
     R.rsp,
     markdown
 VignetteBuilder: R.rsp

NEWS

@@ -1,7 +1,7 @@
 Package: future
 ===============
 
-Version: 1.15.1-9003 [2020-01-02]
+Version: 1.15.1-9003 [2020-01-09]
 
 SIGNIFICANT CHANGES:
 
@@ -26,6 +26,16 @@ NEW FEATURES:
    current RNG state is used.  Use seed = FALSE when it is known that the
    future does not use RNG.
 
+ * Add support for automatically disable multi-threading when using 'multicore'
+   futures. For now, the default is to allow multi-threaded processing but
+   this might change in the future. To disable multi-threaded, set option
+   'future.fork.multithreading.enable' to 'FALSE'. This requires that
+   'RhpcBLASctl' package is installed. Parallelization via multi-threaded
+   processing (done in native code by some packages and externaly library)
+   while at the same time using forked (aka "multicore") parallel processing
+   is known to unstable.  Note that this is not only true when using
+   plan(multicore) but also when using, for instance, parallel::mclapply().
+
 BUG FIXES:
 
  * Evaluation of futures could fail if the global environment contained

R/MulticoreFuture-class.R

@@ -54,6 +54,32 @@ run.MulticoreFuture <- function(future, ...) {
   ## also the one that evaluates/resolves/queries it.
   assertOwner(future)
 
+  ## Disable multi-threading in futures?
+  value <- Sys.getenv("R_FUTURE_FORK_MULTITHREADING_ENABLE", "TRUE")
+  value <- isTRUE(as.logical(value))
+  value <- getOption("future.fork.multithreading.enable", value)
+  if (isFALSE(value)) {
+    if (debug) mdebug("- Evaluate future in single-threaded mode ...")
+    if (!requireNamespace("RhpcBLASctl", quietly = TRUE)) {
+      stop(FutureError(sprintf("In order to disable multi-threading in multicore futures, the %s package must be installed", sQuote("RhpcBLASctl"))))
+    }
+
+    ## Tell OpenMP to use a single thread
+    old_omp_threads <- RhpcBLASctl::omp_get_max_threads()
+    if (old_omp_threads > 1L) {
+      RhpcBLASctl::omp_set_num_threads(1L)
+      on.exit(RhpcBLASctl::omp_set_num_threads(old_omp_threads), add = TRUE)
+      if (debug) mdebug("  - Force single-threaded processing for OpenMP")
+    }
+
+    ## Tell BLAS to use a single thread(?)
+    ## NOTE: Is multi-threaded BLAS an issue? Have we got any reports on this.
+    ## FIXME: How can we get the current BLAS settings?
+    ## /HB 2020-01-09
+    ## RhpcBLASctl::blas_set_num_threads(1L)
+    if (debug) mdebug("- Evaluate future in single-threaded mode ... DONE")
+  }
+
   mcparallel <- importParallel("mcparallel")
 
   expr <- getExpression(future)

R/options.R

@@ -64,6 +64,7 @@
 #'  \item{\option{future.availableWorkers.custom}:}{(function) If set and a function, then this function will be called (without arguments) by [availableWorkers()] where its value, coerced to a character vector, is interpreted as hostnames of available workers.}
 #'
 #'  \item{\option{future.fork.enable}:}{(logical) Enable or disable _forked_ processing.  If `FALSE`, multicore futures becomes sequential futures.  If not specified, this option is set according to environment variable \env{R_FUTURE_FORK_ENABLE}.  If `NA`, or not set (the default), the a set of best-practices rules decide whether should be supported or not.  See [supportsMulticore()] for more details.}
+#'  \item{\option{future.fork.multithreading.enable}:}{(logical) Enable or disable _multi-threading_ while using _forked_ parallel processing.  If `FALSE`, different multi-thread library settings are overridden such that they run in single-thread mode, which requires that the \pkg{RhpcBLASctl} package is installed.  If not specified, this option is set according to environment variable \env{R_FUTURE_FORK_MULTITHREADING_ENABLE}.  If `TRUE`, or not set (the default), multi-threading is allowed.  Parallelization via multi-threaded processing (done in native code by some packages and externaly library) while at the same time using forked (aka "multicore") parallel processing is known to unstable.  Note that this is not only true when using `plan(multicore)` but also when using, for instance, [parallel::mclapply()].}
 #'
 #'  \item{\option{future.supportsMulticore.unstable}:}{(character) Controls whether a warning should be produced or not whenever multicore processing is automatically disabled because the environment in which R runs is considered unstable for forked processing, e.g. in the RStudio environment.  If `"warning"` (default), then an informative warning is produces the first time 'multicore' or 'multiprocess' futures are used.  If `"quiet"`, no warning is produced.  If not specified, this option is set according to environment variable \env{R_FUTURE_SUPPORTSMULTICORE_UNSTABLE}.  See [supportsMulticore()] for more details.}
 #' }