nix: Respect meta.outputsToInstall, and use all outputs by default

[edolstra]

nix profile install will now install all outputs listed in the package's meta.outputsToInstall attribute, or all outputs if that attribute doesn't exist. This makes it behave consistently with nix-env. Fixes #6385.

Furthermore, for consistency, all other nix commands do this as well. E.g. nix build will build and symlink the outputs in
meta.outputsToInstall, defaulting to all outputs. Previously, it only built/symlinked the first output. Note that this means that selecting a specific output using attrpath selection (e.g. nix build nixpkgs#libxml2.dev) no longer works. A subsequent PR will add a way to specify the desired outputs explicitly.

[Ericson2314]

A subsequent PR will add a way to specify the desired outputs explicitly.

Are you saying

nix build nixpkgs#libxml2.dev

will work once again, or there will be an alternative UI?

[edolstra]

I'm thinking using the ! syntax, i.e.

nix build 'nixpkgs#libxml2!dev,man'

Not super-intuitive, but the advantage over --outputs dev,man is that the set of outputs is part of the installable, so

• Different installables can have different outputs, e.g. `nix shell 'nixpkgs#foo!bin' 'nixpkgs#bar!man'.
• nix profile will remember which outputs the user wants to use across profile upgrades.

[Ericson2314]

Well, you certainly know I like the !dev,man part :). But with RFC 92:

nixpkgs#libxml2!dev,man

could mean

nixpkgs#libxml2.outPath!dev,man

or

nixpkgs#libxml2.drvPath!dev,man

You mean the latter, but quite arguably the former is what one would expect if the meaning is given by composition, just as #4543 works. In any event, both are valid things to request with RFC, in case libxml's out produces another derivation.

---

If you check out https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix, notice outputSpecified:

https://github.com/NixOS/nixpkgs/blob/f4acbd402b9790add87e4cf543e0199ea7dc3617/lib/attrsets.nix#L598-L601 https://github.com/NixOS/nixpkgs/blob/f4acbd402b9790add87e4cf543e0199ea7dc3617/lib/customisation.nix#L198-L206

I would propose making that part of Nix proper:

diff --git a/src/libexpr/primops/derivation.nix b/src/libexpr/primops/derivation.nix
index c0fbe8082..4ec79eab2 100644
--- a/src/libexpr/primops/derivation.nix
+++ b/src/libexpr/primops/derivation.nix
@@ -19,6 +19,7 @@ let
         drvPath = strict.drvPath;
         type = "derivation";
         inherit outputName;
+        outputSpecified = true;
       };
     };

and then using it so libxml.out vs libxml will just do what people expect.

[abathur]

! seems like a bit of a UX trap given its role (when not quoted) as the (default) history expansion character in at least bash and zsh.

[edolstra]

The problems with libxml.out and outputSpecified are:

• They don't provide a concise way to refer to multiple outputs, e.g. you can't install the bin and man outputs of a package, except as separate profile elements, which could then diverge (e.g. you could update the bin element and forget to update man).
• I'm not sure what the intended semantics of outputSpecified are, but it would be pretty confusing if meta.outputsToInstall applies to libxml but not to libxml.<output>. It also requires libxml and libxml.<default-output> to be different attrsets, so that requires an additional allocation.

[Ericson2314]

Well, we can discuss on the UI call. I suspect many people will be extremely surprised if libxml.dev is in fact all outputs.

[thufschmitt]

This is a rather big breaking change, isn’t it? Esp. until there’s a way to specify the wanted outputs. It certainly warrants at the very least a changelog entry, and probably also an easy way to get back the old behavior, doesn’t it?

[tomberek]

At first I was confused why the package.dev behavior was changing. So this is my attempt to better understand the possibilities:

current

• package => only out
• package.out => only out
• package.dev => only dev
• package.lib => only lib

option 1

Would it help if we only complied with meta.outputsToInstall in the cases where the outputName is also the first value in outputs? Otherwise allow the current specificity to exist:

• package => meta.outputsToInstall or everything if it doesn't exist
• package.out => only out (using outputSpecified)
• package.dev => only dev
• package.lib => only lib

This matches expectations with the "binaries first" policy. Though this does make it tougher to easily express only package.out. And that seems to be what outputSpecified allows us to do. It comes with a possibility to have mismatches by upgrading p.dev and not p.lib.

option 2

Another possible behavior (with appropriate dedup)

• package => meta.outputsToInstall or everything if it doesn't exist
• package.out => out ++ meta.outputsToInstall
• package.dev => dev ++ meta.outputsToInstall
• package.lib => lib ++ meta.outputsToInstall

This convention would prevent possible mismatch, but does make it much harder to be specific about the intention - so at the moment I don't think this is better or more clear than the previous.

option3 (the PR currently?)

• package => meta.outputsToInstall
• package.out => meta.outputsToInstall
• package.dev => meta.outputsToInstall
• package!out => only out
• package!out,man => out ++ man
• package!dev,lib => dev ++ lib

It is only the more specific behavior that will be impacted and used by more-informed users. The advantage of this over Option 1 is that it allows arbitrary pairing together outputs as desired. The question is if that use-case is common enough to warrant the additional cases, new usage+syntax and documentation/explanation burden? Can we have the simpler usage of package.out or package.dev still support the expected outcome and mean only out or dev respectively with the outputSpecified mechanism?

Is this a conflict with RFC92 syntax or can it re-use the same machinery? (@Ericson2314 seemed to suggest it can be made consistent).

Note: any of these still provide the edge case of mismatched profiles/upgrades with pathological choices, eg: package!dev and package!lib,man as separate elements.

[lovesegfault]

Please don't use ! for this, it's going to be horrible for users, especially those not familiar with bash/zsh who will not understand what's going wrong.

We already have the wonky # syntax which causes all sorts of weirdness with autocompletion, let's not double down on making Nix hard to use.

Could : be used? Maybe @?

[edolstra]

: and @ are not an option because they're highly ambiguous in URLs / flake references. I'm not attached to ! but there are not that many other options. (It is also the syntax we've used for a long time in the old CLI, but that was not really documented.)

[edolstra]

We could use caret (^). It's not special to the shell, doesn't occur often in URLs, and there is some precedence for using it as a separator (e.g. Git's <commit>^<n> syntax).

[lovesegfault]

I think only ^ and * remain that wouldn't trigger shell stuff.

Alternatively, perhaps a cli flag is worth considering? nix build .#foo --out lib or something like that.

[edolstra]

The problem with a CLI flag is that isn't not associated with an installable, so you can't select different outputs per installable, as in

# nix shell 'nixpkgs#foo!bin,man' 'nixpkgs#bar!dev'

[tomberek]

This idea of allowing more specificity and control about the specific outputs is a more advanced feature. Most of initial users will simply rely upon meta.outputsToInstall. After that there are plenty of issues with * and ! in shells. The ^ is an interesting options i had not considered, but the re-use of an existing concept like ! along with RFC92 is elegant and I don't think it is a high price to pay for advanced users to add some quotes. (some flakerefs with query parameters require quotes already)

Conclusion: I think ! would be best if it comes along with RFC92.

[Ericson2314]

The best use of ! I can think of currently with fancier installables, compatible with RFC 92, is requiring foo.drvPath!outputs or foo.outPath!outputs, and banning plain foo!outputs as ambiguous. It is much easier to relax restrictions later than tighten them.

I would still want foo.out to work in ways user expects

It also requires libxml and libxml. to be different attrsets, so that requires an additional allocation.

I am not worried about this because, per the code I linked, Nixpkgs is already doing this. Doing it in Nix instead has no marginal cost in terms of the number of allocations.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-29/18903/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/recent-nix-change-worthy-of-greater-attention/18972/1