Introduce `types.raw` #160489

infinisil · 2022-02-17T17:19:24Z

Motivation for this change

Introduce types.raw as a type that doesn't do any nested processing for mkIf, mkForce, etc. on its values. This is useful when this processing would throw errors or where it would be too expensive (such as with package sets like nixpkgs).
Use types.raw for _module.args's type. This finally fixes _module.args.name gets merged on assignment #53458

This is split off from #132448. This is also motivated by wanting to deprecate types.attrs and types.unspecified, for which types.attrsOf types.raw and types.raw (or types.anything) is an adequate general replacement.

Things done

Wrote and ran tests
Wrote docs

roberth

Not sure.

Is this the only use case?

roberth · 2022-02-17T18:07:30Z

lib/modules.nix

@@ -138,7 +138,7 @@ rec {
            # support for that, in turn it's lazy in its values. This means e.g.
            # a `_module.args.pkgs = import (fetchTarball { ... }) {}` won't
            # start a download when `pkgs` wasn't evaluated.
-            type = types.lazyAttrsOf types.unspecified;
+            type = types.lazyAttrsOf types.raw;


I don't think this is the right trade-off. It's already lazy and when you do access an attr value, like that of pkgs, you'll force it anyway. Checking its type attr before returning it is a cheap and one-time operation.

OfBorg just finished. Allocations are down by 0.0002%

https://github.com/NixOS/nixpkgs/pull/160489/checks?check_run_id=5236802082

So I don't think we should trade the ability to mkForce in _module.args for arguably no benefit.

See #160489 (comment), mkForce and co. still work on the spine

I must have read too much into "raw".
The tests demonstrate it clearly.

Perhaps the docs can clarify that it's not entirely raw?

I guess the confusion comes from #132448 where types.raw originally worked like that, not doing any processing at all, but I changed that after discussing it with you :)

I think mentioning that in the docs is a good idea

nixos/doc/manual/from_md/development/option-types.section.xml

+            type is recommended for options whose only value is defined
+            by the same module as the option, because its structure is


infinisil · 2022-02-17T18:46:51Z

@roberth This type still works with mkForce, mkIf and co., just not deeply. E.g. this works and gives a result of 30:

let lib = import ./lib; in

lib.evalModules {
  modules = [
    {
      options.foo = lib.mkOption {
        type = lib.types.raw;
      };

      config.foo = lib.mkMerge [
        (lib.mkIf false 10)
        20
        (lib.mkIf true (lib.mkForce 30))
      ];
    }
  ];
}

types.anything is similar to types.raw in that it accepts any value, however types.anything recurses deeply into the structure to apply any mkForce and co. And that is what makes types.anything very unsuitable for below-described use cases.

The main use case for types.raw is values that aren't meant for the module system to be evaluated and merged, because they come from outside the module system. So e.g. things like package sets, which have their own merging strategies incompatible with the module system: overlays, overrides, extends, etc. The module system also shouldn't try to inspect these values for any mkForces and co, which would make it excessively strict in evaluation without any benefit, since you won't be using mkForce and co. for e.g. pkgs.hello

Other than _module.args I can also think of nixpkgs.pkgs and options like services.xserver.windowManager.xmonad.haskellPackages that could benefit from this.

We also need to deprecate types.unspecified due to its implicit and arbitrary merging behavior, and replace it with either types.anything or types.raw.

roberth · 2022-02-17T18:49:08Z

Is this the only use case?

This one seems valid.

#157056 (comment)

roberth · 2022-02-17T19:03:30Z

nixos/doc/manual/development/option-types.section.md

+    recommended for options whose only value is defined by the same module
+    as the option, because its structure is known ahead-of time and doesn't
+    need to be typechecked internally. Especially useful for values where
+    type checking would be too expensive, or where type checking is handled
+    in another way.


I think what's important is that

it doesn't merge or check

it does apply priorities

it should be used if and only if merging and checking are not desirable

"Especially useful" and "recommended" are a bit too encouraging for something that should be applied with attention.

Sounds good, I updated the docs

Fixes NixOS#53458, as types.raw doesn't allow setting multiple values

infinisil · 2022-02-22T14:56:40Z

nixos/doc/manual/development/option-types.section.md

+:   A type which doesn't do any checking, merging or nested evaluation. It
+    accepts a single arbitrary value that is not recursed into, making it
+    useful for values coming from outside the module system, such as package
+    sets or arbitrary data. Options of this type are still evaluated according
+    to priorities and conditionals, so `mkForce`, `mkIf` and co. still work on
+    the option value itself, but not for any value nested within it. This type
+    should only be used when checking, merging and nested evaluation are not
+    desirable.


@roberth Does this sound good?

ncfavier · 2022-02-24T09:01:13Z

As expected, this broke my config because I was adding my own stuff to _module.args.utils. What was less expected is that instead of a module system error I got an infinite recursion (logs).

Probably just a "me" problem not worth looking into as I couldn't reproduce that error on a minimal config.

roberth · 2022-02-24T10:09:17Z

I'd like utils to be removed, as it is nonsensical.

ncfavier · 2022-02-24T10:21:13Z

Yeah, I guess it's supposed to be a middle ground between lib and pkgs? Maybe the things it defines should go into config.lib instead?

roberth · 2022-02-24T11:52:20Z

As a general rule, utils is the name for the place people create to put the stuff they don't know where to put. config.lib is only slightly better.

The systemd logic can move into importable modules instead, after #156533

nixos-discourse · 2022-03-04T13:38:36Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-25/18003/1

infinisil added the 6.topic: module system About "NixOS" module system internals label Feb 17, 2022

infinisil requested a review from roberth February 17, 2022 17:19

infinisil requested review from edolstra and nbp as code owners February 17, 2022 17:19

infinisil mentioned this pull request Feb 17, 2022

Introduce types.raw and types.unconditional #132448

Closed

7 tasks

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation This PR adds or changes documentation labels Feb 17, 2022

infinisil mentioned this pull request Feb 17, 2022

Make lib more unified and overridable #157056

Closed

infinisil force-pushed the types.raw branch from a1a1ece to ff958c7 Compare February 17, 2022 17:50

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Feb 17, 2022

roberth reviewed Feb 17, 2022

View reviewed changes

roberth mentioned this pull request Feb 17, 2022

_module.args.name gets merged on assignment #53458

Closed

roberth reviewed Feb 17, 2022

View reviewed changes

infinisil force-pushed the types.raw branch from ff958c7 to 53cd7bf Compare February 17, 2022 19:41

infinisil added 2 commits February 22, 2022 15:54

lib/types: Introduce types.raw for unprocessed values

665344f

lib/modules: Use types.raw for _module.args

ff13cd5

Fixes NixOS#53458, as types.raw doesn't allow setting multiple values

infinisil force-pushed the types.raw branch from 53cd7bf to ff13cd5 Compare February 22, 2022 14:54

infinisil commented Feb 22, 2022

View reviewed changes

roberth approved these changes Feb 23, 2022

View reviewed changes

roberth merged commit 6225804 into NixOS:master Feb 23, 2022

infinisil deleted the types.raw branch February 23, 2022 17:39

infinisil mentioned this pull request Mar 16, 2022

Build fails with nix-darwin: The option `_module.args.inputs' is defined multiple times. gytis-ivaskevicius/flake-utils-plus#113

Closed

ncfavier mentioned this pull request Mar 21, 2022

nixos: replace types.anything with submodule type nix-community/home-manager#2396

Merged

7 tasks

ncfavier mentioned this pull request Jul 23, 2023

nixos-generate-config: add 'lib' arg to configuration.nix #244653

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `types.raw` #160489

Introduce `types.raw` #160489

infinisil commented Feb 17, 2022

roberth left a comment

roberth Feb 17, 2022

roberth Feb 17, 2022 •

edited

Loading

roberth Feb 17, 2022

infinisil Feb 17, 2022

roberth Feb 17, 2022

infinisil Feb 17, 2022

This comment was marked as outdated.

infinisil commented Feb 17, 2022 •

edited

Loading

roberth commented Feb 17, 2022

roberth Feb 17, 2022

infinisil Feb 17, 2022

infinisil Feb 22, 2022

ncfavier commented Feb 24, 2022

roberth commented Feb 24, 2022

ncfavier commented Feb 24, 2022

roberth commented Feb 24, 2022

nixos-discourse commented Mar 4, 2022

		type is recommended for options whose only value is defined
		by the same module as the option, because its structure is

Introduce types.raw #160489

Introduce types.raw #160489

Conversation

infinisil commented Feb 17, 2022

Motivation for this change

Things done

roberth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roberth Feb 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

infinisil commented Feb 17, 2022 • edited Loading

roberth commented Feb 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncfavier commented Feb 24, 2022

roberth commented Feb 24, 2022

ncfavier commented Feb 24, 2022

roberth commented Feb 24, 2022

nixos-discourse commented Mar 4, 2022

Introduce `types.raw` #160489

Introduce `types.raw` #160489

roberth Feb 17, 2022 •

edited

Loading

infinisil commented Feb 17, 2022 •

edited

Loading