Consider removing or optionalizing the gc() in this while loop

[kendonB]

It looks like the vast majority of the overhead time causing the problem in #449 is spent in this while loop:

https://github.com/ropensci/drake/blob/master/R/mclapply.R#L75-L78

If I remove the gc() I see substantial speed improvements, though the master process certainly still can't keep up with the workers for the up-to-date targets which obviously process fast. A better solution would be to do a round of preprocessing targets using the same strategy as outdated so that the first round of targets sent in fl_master are out-of-date.

[wlandau]

cc @krlmlr. Ref: comment, #428, #429. Looks like we should use gc() more sparingly.

[wlandau]

And since persistent workers check if targets can be skipped, this may be yet another issue improved with #440.

[wlandau]

In de5835f, drake calls gc() after each target build and on exit. Please see how it works for you. We can always elect to call gc() less often, but I think de5835f may strike a nice compromise.

[kendonB]

@wlandau de5835f is going to be even worse as it will call gc() for every up-to-date target.

[wlandau]

Then 2002653 should be an improvement.

[kendonB]

This is surely still going to be too much gc'ing for a lot of projects. gc'ing often takes 2-3 seconds and for projects with small targets this will add a lot of overhead.

Perhaps your original solution could be modified to detect when a worker is idle for 30 seconds and only gc under those conditions and only once per idling event. Now that I've thought about it this seems like the obvious answer.

[kendonB]

Big improvements with 2002653 using future_lapply. Still took more than 25 mins to get through the up-to-date targets but I'm now I've been staring at the logs waiting for "target ..." to appear for 5 minutes.