nixos: set default for boot.tmpOnTmpfs to true

[cmfwyp]

Motivation for this change

Mounting /tmp as a tmpfs keeps /tmp clean on boot and diminishes disk usage, which in turn can save power and increase lifetime of SSD devices. It is the default behavior on Fedora, Arch Linux, and Ubuntu.

Things done

• Tested on NixOS with the nixos-unstable channel.
• Fits CONTRIBUTING.md.

---

This currently uses the tmp.mount systemd unit, which has issues #14777 and #10670. It is perhaps preferable to wait for #14778 to be merged so that these issues are solved.

Since this is an incompatible change, I added an item to the release notes for 16.09.

[mention-bot]

@cmfwyp, thanks for your PR! By analyzing the annotation information on this pull request, we identified @ehmry, @domenkozar and @vcunat to be potential reviewers

[abbradar]

One concern that I have is that building massive projects like Chromium can eat all the RAM with tmpfs because Nix creates build directories in /tmp. My proposed solution is to change that to /var/tmp.

[edolstra]

Nitpick: Shouldn't this option be named boot.tmpfsOnTmp (as it mounts a tmpfs on /tmp, not /tmp on a tmpfs).

[cmfwyp]

Yes, that would be the correct option name. I can push a commit to rename it to that and alias the old name.

@abbradar Good point. Builds are something that would preferably be done in memory, to make them faster and use the disk less, but systems may not have a swap file and therefore come out of memory pretty fast with some builds, which is a more important concern.

On the other hand, I think this would already be an issue when using Nix on most Linux distributions that mount a tmpfs on /tmp; which can mean either that this is already a problem on these distributions and that the path should be changed to /var/tmp (how likely is changing the default path for Nix to break things?), or that it has not been a problem on these distributions and as a consequence that it should not be a problem for NixOS either.

/tmp is the more semantically-correct place for builds, since they do not need to persist for a long time or even through reboots, but it is not ideal for large files.¹

1: A good comparison: http://0pointer.de/blog/projects/tmp.html.

[abbradar]

@cmfwyp Subjectively that would be a bigger problem on NixOS because it's the main platform for overall Nix development and also for Hydra hosts, which together (anecdotally!) account for almost all "big" builds in the Linux/Nix world.

That said, I also do like /tmp more for storing builds precisely because of its FHS semantics (they are not needed to be "persistent"). Also, speed and disk wear concerns for small builds are important too.

I'm not sure how this problem should be solved. For some prior experience, on Arch Linux /tmp is used for builds by yaourt which is a popular tool for building and installing packages from source. I needed to remount /tmp or move build to somewhere enough times for me to remember (but not really frequently) when I had 2 and later 4GB of RAM.

[romildo]

In Gentoo Linux I usually mount /tmp as a tmpfs file system, and it is used for building most of the packages. For particular packages which demand too much space I set a per package environment variable PORTAGE_TMPDIR which is used to build that package.

Maybe something in this line can be figured out for NixOS too.

[cmfwyp]

The more common name seems to be "tmp on tmpfs" ([1] [2] [3] [4]), so I suggest keeping that.

Regarding large builds, I see multiple options. Adding a meta-attribute for build expressions that use large files is not an elegant solution. Build servers (Hydra or otherwise) that do builds with large files can set boot.tmpOnTmpfs to false. Ideally, there would also be a way with nix-build to perform the build in /var/tmp instead of /tmp, which could be used when developing for builds that use large files.

Builds could also all be done in /var/tmp, but /tmp is the semantically correct location and using a tmpfs for builds makes sense given that Nix systematically removes the build folder once the build is done. Doing builds using a tmpfs should make them faster since it will download and write files to volatile memory instead of disk. Reads are already usually cached by the kernel when there is memory available.

[fpletz]

Deferred to 17.09 because a solution to use another directory on a per build basis would be needed.

[abbradar]

Another solution may be to make Nix build in /var/tmp instead.

[ckauhaus]

Any news on this one? Will this make it into 18.03?

[samueldr]

Adopting into 18.09 not for definitive inclusion, but since it previously was part of release milestones.

---

I would like to know if the previous concerns are still relevant, and what work is being done to fix them, if any.

---

And while I was sending this, something related popped-up

[18:54:47] <symphorien> Nix is already ram hungry enough that I wouldn't like some ram to be used for /tmp
[18:55:46] <infinisil> Good point heh

• https://logs.nix.samueldr.com/nixos/2018-08-02#1533250370-1533250311;

This may be a valid concern, with the previously shared concerns.

[vcunat]

It's a rather special use case, but I disabled this on the t4{a,b} machines on Hydra. That certainly lead to some slight slow-down, but reliability increased. The machines don't have that little RAM (2-3 GiB/thread) and "big-parallel" builds didn't go to them, but some problems still happened daily.

[infinisil]

18:45 <infinisil> samueldr: I'm removing #17494 from the milestone, that alright? Nobody seems to really care much about it
18:45 <{^_^}> https://github.com/NixOS/nixpkgs/pull/17494 (by cmfwyp, 2 years ago, open): nixos: set default for boot.tmpOnTmpfs to true
18:48 <samueldr> I think a definitive action (merge / close) might need to happen with regards to the question, rather than moving it on the side
18:50 <samueldr> and with the discussion happpening in the PR, and previous discussions on different nixos irc channels, I think that it should be closed; disk space being a cheaper resource to abuse than working memory (RAM) means it probably is better to keep the current default
18:50 <samueldr> *especially* considering how not all FSes can receive a swap file
18:56 <gchristensen> +1
18:56 <gchristensen> but also kinda wish deleting temp on boot was default
18:56 <samueldr> that's probably the more right alternative to the same intents
18:57 <samueldr> though I was thinking it might need a "once you're installed" section to an installation appendix, with common configuration stories
18:57 <gchristensen> a GREAT idea
18:58 <infinisil> Alright I'll close it