make via set_progress fails on Windows with long filename

[kendonB]

ref: richfitz/storr#94

I'm finding that drake throws an error from within set_progress when making. Reprex on Windows:

library(drake)
fn <- paste0(paste0(rep("a", 150), collapse = ""), ".rds")
saveRDS(iris, fn)
on.exit(file.remove(fn))
plan <- drake_plan(my_iris = readRDS(file_in("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.rds")),
                   strings_in_dots = "literals")
make(plan)
#> Error in file.rename(tmp, filename): expanded 'to' name too long

[kendonB]

A workaround:

library(drake)
fn <- paste0(paste0(rep("a", 150), collapse = ""), ".rds")
saveRDS(iris, fn)
plan <- drake_plan(my_fn = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.rds",
                   my_iris = readRDS(file_in(my_fn)),
                   strings_in_dots = "literals")
make(plan)
#> target my_fn
#> target my_iris
file.remove(fn)

^{Created on 2018-12-19 by the reprex package (v0.2.1.9000)}

[wlandau]

You can also turn off key mangling in storr, which will also make drake run faster. The only caveat is that you can no longer use file paths with slashes for file_in(), etc.

library(drake)
fn <- paste0(paste0(rep("a", 150), collapse = ""), ".rds")
saveRDS(iris, fn)
on.exit(file.remove(fn))
plan <- drake_plan(my_iris = readRDS(file_in("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.rds")),
                   strings_in_dots = "literals")

cache <- storr::storr_rds(".drake", mangle_key = FALSE)

make(plan, cache = cache)

Related: #572, richfitz/storr#88, and richfitz/storr#90.

It would be nice to avoid key mangling altogether, but files are tricky. Without encoding, a plan with file_in("not/a/valid/key.txt") would error out. We really do need a reliable bijective transformation for the file names.

[wlandau]

As a default option, I believe we will continue to need a bijective function to encode file names, and the solution needs to be general. Otherwise, we will either have bad keys or name collisions. Please chime in if you find an alternative to base 64 encoding that produces shorter file names.