Document what Nix *is* #6420
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,79 @@ | ||||||||
| # Architecture | ||||||||
|
|
||||||||
| *(This chapter is unstable and a work in progress. Incoming links may rot.)* | ||||||||
|
|
||||||||
| This chapter describes how Nix works. | ||||||||
| It should help users understand why Nix behaves as it does, and it should help developers understand how to modify Nix and how to write similar tools. | ||||||||
|
There was a problem hiding this comment. I don't love this introduction although I know it is lifted from the Gazelle documentation. I, at least personally, feel like the concepts described are essential to using Nix/NixOS/Nixpkgs as a developer. Maybe I am over-estimating the difficulty a little, but I think if you want to do anything more with Nix than have a neat roll-back-able package manager, understanding the basic concepts that this document provides is critical. If you want to start hacking on Nix itself or take inspiration for your own tools, I think there's a lot more to know. My opinion is that this introduction might push people away. I think it is good for all Nix users, not just the curious. Perhaps we could add a bit in the middle "developers understand how to leverage Nix", then replace the existing "developers" with "those interested", or remove that last bit entirely. To me, this document is about using and understanding Nix and its tooling and ecosystem rather than the gory details of its internal operation. Like you'd read this to understand how to modify Nixpkgs and how to write your own similar expressions, instead of how to write your own build system programming language. Hopefully this makes sense. There was a problem hiding this comment.
I thought so, too. After discussing with experienced people a lot, and doing studies with actual users, I have to disagree. We instead need better learning material, and this is why we will put work into nix.dev. I'm still convinced we have to front-load learners with specific knowledge instead of handing them inscrutable examples and templates, as Nix is different from what they usually know. But it has to be the right things, and user-oriented knowledge is different from the conceptual background presented here.
There should be some very condensed variant of this in the nix.dev guides. Right now it's not a priority for me. Would you be interested in giving it a try? The depth presented here should also be available on the onboarding path, but clearly optional. I'm convinced it's not really needed to use Nix effectively, but eventually studies will have to answer that question.
Yes, absolutely. This architecture chapter should be the highest level of abstraction to start with. Eventually we should have paths into the source code, with additional prose sections discussing more detailed concepts, and then linking to the code to help discovering the implementation. |
||||||||
|
|
||||||||
| ## Overview | ||||||||
|
|
||||||||
| Nix consists of [hierarchical layers][layer-architecture]. | ||||||||
|
|
||||||||
| ``` | ||||||||
| +-----------------------------------------------------------------+ | ||||||||
| | Nix | | ||||||||
|
There was a problem hiding this comment.
Suggested change
Tbh this feels a bit absurd, but where do we draw the line of what terminology is "too specific" for the introduction? By the same reasoning, we should perhaps replace There was a problem hiding this comment.
Exactly, this is why it is that way and nothing else.
We use "evaluate" because that is what pure languages do. That term is also understood by people not into programming language theory, as it is self-explanatory - evaluation produces a value. |
||||||||
| | [ commmand line interface ]------, | | ||||||||
| | | | | | ||||||||
| | evaluates | | | ||||||||
| | | manages | | ||||||||
| | V | | | ||||||||
| | [ configuration language ] | | | ||||||||
| | | | | | ||||||||
| | +-----------------------------|-------------------V-----------+ | | ||||||||
| | | store evaluates to | | | ||||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Maybe we should rather rename the "persistence layer" in the table to "store"? |
||||||||
| | | | | | | ||||||||
| | | referenced by V builds | | | ||||||||
| | | [ build input ] ---> [ build plan ] ---> [ build result ] | | | ||||||||
|
There was a problem hiding this comment.
Suggested change
Isn't "output" the technical term for this, or should "result" encompass more than just the output? If so, what? We should have a small glossary to which we then limit our use of terminology. I wish markdown has There was a problem hiding this comment.
It's the Nix term for this, as is derivation for what we call build task here. We deliberately use generic terms in the overview, as none of that is specific to Nix.
We have that as the Rosetta Stone. The idea is to introduce Nix specific terminology in the subsequent sections, where we also describe how Nix implements the concepts in principle. We could move the table to the overview to accommodate newcomers and experienced Nix users, see #6420 (comment). There was a problem hiding this comment.
Then
This doesn't scale to encompass terminology that is specific to Nix, so we should still have a glossary. Would it make sense to have a translation of the entire diagram? Seems like a good way to learn the new terminology. There was a problem hiding this comment. I do find the duplication of terminology very concerning. There was a problem hiding this comment.
Sounds reasonable, please add a suggestion, considering the following:
I hope this would be covered by additional sections, see the colored outline comment above. Is there anything that would still be missing?
Would take too much head space for my taste. Or maybe just link to the respective pages behind the table, not mentioning it in the table of contents? Sound like detail work though, not relevant for this PR. There was a problem hiding this comment. Humans are really good at memorizing things in places, so I would guess that familiarizing oneself with the general diagram and then learning the technical terms in the same "place" would be highly effective. (I'm not a cognitive psychologist though) You'd have to put them close together for this correspondence to be noticed; not one at the start and one at the end.
One could argue that the latter is out of scope for this document, but in practice, all terminology throughout the ecosystem shares the same namespace. We should recognize that and have a single place to look all of them up, and disambiguate them where necessary. (e.g. "input", which is already conflicting with a term used in this chapter)
The context would imply this, but it'd be helpful to have a caption such as "An overview of the Nix architecture using generic terminology", which then contrasts well with "An overview of the Nix architecture using its specific terms". There was a problem hiding this comment.
We have that in the table of contents, and it is entirely Nix-specific.
Agreed on potential confusion. Suggestions to do anything about that except always referring to "build input" in generic terms, because that is different from just "input"?
I agree with the learning conditions premise, but cannot picture a specific change. Feel free to make a pull request on the fork against this branch. |
||||||||
| | | | | | ||||||||
| | +-------------------------------------------------------------+ | | ||||||||
| +-----------------------------------------------------------------+ | ||||||||
| ``` | ||||||||
|
|
||||||||
| At the top is the [command line interface](../command-ref/command-ref.md), translating from invocations of Nix executables to interactions with the underlying layers. | ||||||||
|
|
||||||||
| Below that is the [Nix expression language](../expressions/expression-language.md), a [purely functional][purely-functional-programming] configuration language. | ||||||||
|
Comment on lines
+32
to
+34
There was a problem hiding this comment. When I started thinking about the term "Nix", this is the hierarchy that helped me put things into perspective: |
||||||||
| It is used to compose expressions which ultimately evaluate to self-contained *build plans*, used to derive *build results* from referenced *build inputs*. | ||||||||
|
There was a problem hiding this comment. As someone who has managed to wrap their head around Nix's confusing usage of terminology, I wonder if a pointer here to the rosetta stone would be good, marked for experienced Nix users. It is something I referred to several times while writing this review to make sure my understanding was correct, as this guide (for good reason) uses terms I'm unfamiliar with. Maybe it should be There was a problem hiding this comment. Agreed with the problem. I want to keep this essential section as concise as possible, though. Therefore definitely will not add variants in parentheses. @tpwrules @Ericson2314 What about moving the Rosetta Stone to the end of the overview section? |
||||||||
|
|
||||||||
| The command line and Nix language are what users interact with most. | ||||||||
|
|
||||||||
| > **Note** | ||||||||
| > The Nix language itself does not have a notion of *packages* or *configurations*. | ||||||||
| > As far as we are concerned here, the inputs and results of a build plan are just data. | ||||||||
|
|
||||||||
| Underlying these is the [Nix store](./store/store.md), a mechanism to keep track of build plans, data, and references between them. | ||||||||
| It can also execute build plans to produce new data. | ||||||||
|
|
||||||||
| A build plan is a series of *build tasks*. | ||||||||
| Each build task has a special build input which is used as *build instructions*. | ||||||||
| The result of a build task can be input to another build task. | ||||||||
|
|
||||||||
| ``` | ||||||||
| +-----------------------------------------------------------------------------------------+ | ||||||||
| | store | | ||||||||
| | ................................................. | | ||||||||
| | : build plan : | | ||||||||
| | : : | | ||||||||
| | [ build input ]-----instructions-, : | | ||||||||
| | : | : | | ||||||||
| | : v : | | ||||||||
| | [ build input ]----------->[ build task ]--instructions-, : | | ||||||||
| | : | : | | ||||||||
| | : | : | | ||||||||
| | : v : | | ||||||||
| | : [ build task ]----->[ build result ] | | ||||||||
| | [ build input ]-----instructions-, ^ : | | ||||||||
| | : | | : | | ||||||||
| | : v | : | | ||||||||
| | [ build input ]----------->[ build task ]---------------' : | | ||||||||
| | : ^ : | | ||||||||
| | : | : | | ||||||||
| | [ build input ]------------------' : | | ||||||||
| | : : | | ||||||||
| | : : | | ||||||||
| | :...............................................: | | ||||||||
| | | | ||||||||
| +-----------------------------------------------------------------------------------------+ | ||||||||
| ``` | ||||||||
|
|
||||||||
| [layer-architecture]: https://en.m.wikipedia.org/wiki/Multitier_architecture#Layers | ||||||||
| [purely-functional-programming]: https://en.m.wikipedia.org/wiki/Purely_functional_programming | ||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| # File System Object | ||
|
There was a problem hiding this comment. Is it intended that this file isn't directly reachable from the outline? |
||
|
|
||
| The Nix store uses a simple file system model for the data it holds in [store objects](store.md#store-object). | ||
|
|
||
| Every file system object is one of the following: | ||
|
|
||
| - File: an executable flag, and arbitrary data for contents | ||
| - Directory: mapping of names to child file system objects | ||
| - [Symbolic link][symlink]: may point anywhere. | ||
|
|
||
| We call a store object's outermost file system object the *root*. | ||
|
|
||
| data FileSystemObject | ||
| = File { isExecutable :: Bool, contents :: Bytes } | ||
| | Directory { entries :: Map FileName FileSystemObject } | ||
| | SymLink { target :: Path } | ||
|
|
||
|
There was a problem hiding this comment. It should be clarified somewhere here that the model does not conceptualize hard links, although they can legally appear in the file system representation of a store. Each FSO which is not the root has exactly one parent and one name. |
||
| Examples: | ||
|
|
||
| - a directory with contents | ||
|
|
||
| /nix/store/<hash>-hello-2.10 | ||
| ├── bin | ||
| │ └── hello | ||
| └── share | ||
| ├── info | ||
| │ └── hello.info | ||
| └── man | ||
| └── man1 | ||
| └── hello.1.gz | ||
|
|
||
| - a directory with relative symlink and other contents | ||
|
|
||
| /nix/store/<hash>-go-1.16.9 | ||
| ├── bin -> share/go/bin | ||
| ├── nix-support/ | ||
| └── share/ | ||
|
|
||
| - a directory with absolute symlink | ||
|
|
||
| /nix/store/d3k...-nodejs | ||
| └── nix_node -> /nix/store/f20...-nodejs-10.24. | ||
|
|
||
| A bare file or symlink can be a root file system object. | ||
| Examples: | ||
|
|
||
| /nix/store/<hash>-hello-2.10.tar.gz | ||
|
|
||
| /nix/store/4j5...-pkg-config-wrapper-0.29.2-doc -> /nix/store/i99...-pkg-config-0.29.2-doc | ||
|
|
||
| Symlinks pointing outside of their own root or to a store object without a matching reference are allowed, but might not function as intended. | ||
| Examples: | ||
|
|
||
| - an arbitrarily symlinked file may change or not exist at all | ||
|
|
||
| /nix/store/<hash>-foo | ||
| └── foo -> /home/foo | ||
|
|
||
| - if a symlink to a store path was not automatically created by Nix, it may be invalid or get invalidated when the store object is deleted | ||
|
There was a problem hiding this comment. Not sure what "automatically created by Nix" means here. Do you mean GC roots? This might confuse derivation writers as unless you play nasty games with string contexts or randomly choose a store path out of thin air, creating a symlink in a derivation using a simple There was a problem hiding this comment. "Automatically" means through Nix mechanisms. Both will render to the same file system object: this will not create a reference this will create a reference See #4280 (comment) @tpwrules Do you have a suggestion to put it concisely and be more precise? |
||
|
|
||
| /nix/store/<hash>-bar | ||
| └── bar -> /nix/store/abc...-foo | ||
|
|
||
| Nix file system objects do not support [hard links][hardlink]: | ||
| each file system object which is not the root has exactly one parent and one name. | ||
| However, as store objects are immutable, an underlying file system can use hard links for optimization. | ||
|
|
||
| [symlink]: https://en.m.wikipedia.org/wiki/Symbolic_link | ||
| [hardlink]: https://en.m.wikipedia.org/wiki/Hard_link | ||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,105 @@ | ||||||
| # Store Path | ||||||
|
|
||||||
| Nix implements [references](store.md#reference) to [store objects](store.md#store-object) as *store paths*. | ||||||
|
|
||||||
| Store paths are pairs of | ||||||
|
|
||||||
| - a 20-byte [digest](#digest) for identification | ||||||
| - a symbolic name for people to read. | ||||||
|
|
||||||
| Example: | ||||||
|
|
||||||
| - digest: `b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z` | ||||||
| - name: `firefox-33.1` | ||||||
|
|
||||||
| It is rendered to a file system path as the concatenation of | ||||||
|
|
||||||
| - [store directory](#store-directory) | ||||||
| - path-separator (`/`) | ||||||
| - [digest](#digest) rendered in a custom variant of [base-32](https://en.m.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters) | ||||||
| - hyphen (`-`) | ||||||
| - name | ||||||
|
|
||||||
| Example: | ||||||
|
|
||||||
| /nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1 | ||||||
| |--------| |------------------------------| |----------| | ||||||
| store directory digest name | ||||||
|
|
||||||
| ## Store Directory | ||||||
|
|
||||||
| Every [store](./store.md) has a store directory. | ||||||
|
|
||||||
| If the store has a [file system representation](./store.md#files-and-processes), this directory contains the store’s [file system objects](#file-system-object), which can be addressed by [store paths](#store-path). | ||||||
|
|
||||||
| This means a store path is not just derived from the referenced store object itself, but depends on the store the store object is in. | ||||||
|
|
||||||
| > **Note** | ||||||
| > The store directory defaults to `/nix/store`, but is in principle arbitrary. | ||||||
|
|
||||||
| It is important which store a given store object belongs to: | ||||||
| Files in the store object can contain store paths, and processes may read these paths. | ||||||
| Nix can only guarantee [referential integrity](store/closure.md) if store paths do not cross store boundaries. | ||||||
|
|
||||||
| Therefore one can only copy store objects to a different store if | ||||||
|
|
||||||
| - the source and target stores' directories match | ||||||
|
|
||||||
| or | ||||||
|
|
||||||
| - the store object in question has no references, that is, contains no store paths. | ||||||
|
|
||||||
| One cannot copy a store object to a store with a different store directory. | ||||||
| Instead, it has to be rebuilt, together with all its dependencies. | ||||||
| It is in general not enough to replace the store directory string in file contents, as this may render executables unusable by invalidating their internal offsets or checksums. | ||||||
|
There was a problem hiding this comment.
Suggested change
I don't like the mention of checksums here because Nix already can and does replace store paths with equal-length ones (for content-addressing and There was a problem hiding this comment. I tried to encode exactly this without going into too much detail. This is why it says "in general" and "may render unusable". It is still worth mentioning the checksums, because even if the trouble is rare, it's real and conceptually built in. |
||||||
|
|
||||||
| # Digest | ||||||
|
|
||||||
| In a [store path](#store-path), the [digest][digest] is the output of a [cryptographic hash function][hash] of either all *inputs* involved in building the referenced store object or its actual *contents*. | ||||||
|
|
||||||
| Store objects are therefore said to be either [input-addressed](#input-addressing) or [content-addressed](#content-addressing). | ||||||
|
|
||||||
| > **Historical Note** | ||||||
| > The 20 byte restriction is because originally digests were [SHA-1][sha-1] hashes. | ||||||
| > Nix now uses [SHA-256][sha-256], and longer hashes are still reduced to 20 bytes for compatibility. | ||||||
|
|
||||||
| [digest]: https://en.m.wiktionary.org/wiki/digest#Noun | ||||||
| [hash]: https://en.m.wikipedia.org/wiki/Cryptographic_hash_function | ||||||
| [sha-1]: https://en.m.wikipedia.org/wiki/SHA-1 | ||||||
| [sha-256]: https://en.m.wikipedia.org/wiki/SHA-256 | ||||||
|
|
||||||
| ### Reference scanning | ||||||
|
|
||||||
| When a new store object is built, Nix scans its file contents for store paths to construct its set of references. | ||||||
|
|
||||||
| The special format of a store path's [digest](#digest) allows reliably detecting it among arbitrary data. | ||||||
| Nix uses the [closure](store.md#closure) of build inputs to derive the list of allowed store paths, to avoid false positives. | ||||||
|
|
||||||
| This way, scanning files captures run time dependencies without the user having to declare them explicitly. | ||||||
| Doing it at build time and persisting references in the store object avoids repeating this time-consuming operation. | ||||||
|
|
||||||
| > **Note** | ||||||
| > In practice, it is sometimes still necessary for users to declare certain dependencies explicitly, if they are to be preserved in the build result's closure. | ||||||
| This depends on the specifics of the software to build and run. | ||||||
| > | ||||||
| > For example, Java programs are compressed after compilation, which obfuscates any store paths they may refer to and prevents Nix from automatically detecting them. | ||||||
|
|
||||||
| ## Input Addressing | ||||||
|
|
||||||
| Input addressing means that the digest derives from how the store object was produced, namely its build inputs and build plan. | ||||||
|
|
||||||
| To compute the hash of a store object one needs a deterministic serialisation, i.e., a binary string representation which only changes if the store object changes. | ||||||
|
|
||||||
| Nix has a custom serialisation format called Nix Archive (NAR) | ||||||
|
There was a problem hiding this comment. As an aside, it would be really nice to document NAR's binary format somewhere else than the PDF of the thesis. There was a problem hiding this comment. Yes, absolutely. I had typed out the algorithm in Haskell syntax at some point (haven't checked against the code if it's still exactly the same), but we're not at the level of detail here. Feel free to open a pull request! serialise fso = str "nix-archive-1" ++ serialize' fso
serialise' fso = str "(" ++ serialise'' fso ++ str ")"
serialise'' (File isExecutable contents) =
str "type" ++ str "regular" ++ exec ++ str "contents" ++ str contents
where exec = if isExecutable
then str "executable" ++ str ""
else str ""
serialise'' (SymLink target) =
str "type" ++ str "symlink" ++ str "target" ++ str target
serialise'' (Directory entries) =
str "type" ++ str "directory" ++ concatMap serialiseEntry (sort entries)
where serialiseEntry name fso =
str "entry" ++ str "("
++ str "name" ++ str name
++ str "node" ++ serialise' fso
++ str ")"
str s = int (length s) ++ pad s
int n = _ -- 64-bit little-endian representation of number `n`
pad s = _ -- byte sequence `s`, padded with 0s to a multiple of 8 bytes |
||||||
|
|
||||||
| Store object references of this sort can *not* be validated from the content of the store object. | ||||||
| Rather, a cryptographic signature has to be used to indicate that someone is vouching for the store object really being produced from a build plan with that digest. | ||||||
|
|
||||||
| ## Content Addressing | ||||||
|
|
||||||
| Content addressing means that the digest derives from the store object's contents, namely its file system objects and references. | ||||||
| If one knows content addressing was used, one can recalculate the reference and thus verify the store object. | ||||||
|
|
||||||
|
There was a problem hiding this comment. Slight bit of an aside, but one thing I never understood about content addressing: what says that I actually used a given build plan to produce a store object with this digest? Don't you still need someone to vouch for the fact that this build plan produced that output? Or is the content-addressed digest also a function of the input digest? There was a problem hiding this comment. @Ericson2314 told me there is a database entry in the store that maps content hashes to input hashes. Appears to be a not well-known feature. I suppose reproducing the build would verify the mapping, but that's obviously not guaranteed to succeed, as the build may not be bytewise reproducible. This is an important detail, but not at this stage. See #6420 (comment) for the outline I envision. The digests should have their own chapter with more structure and details. Please open a pull request if you want to tackle that. |
||||||
| Content addressing is currently only used for the special cases of source files and "fixed-output derivations", where the contents of a store object are known in advance. | ||||||
| Content addressing of build results is still an [experimental feature subject to some restrictions](https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md). | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This may sound like splitting hairs, but instead of the word "Nix", the manuals should say explicitly what the term refers to, even if it is to the detriment of brevity.1 Depending on the context, I saw it used to mean:
NixOS/nix)NixOS/niximplementation, Nixpkgs, NixOS, even NixOps)Taking the very next sentence as an example:
I guess the "Architecture" section is about the model itself, and if so, it would be prudent to start with a short explanation what it is about. (It would be nice to have a specification.) Below are two ways I tried to define "Nix model" to myself:
A purely functional deployment model
... then explain what purity2, "functional" (this from PR Greatly expand architecture section, including splitting into abstract vs concrete model #6877), and deployment model mean.
(Took it directly from the thesis; the omission of the word "software" is deliberate.)
A (new) paradigm of building anything that requires assembly3, and managing the build results. The official term for "build result" is store object because they are kept in the Nix store.
[1]: The definition of "Nix" has been a subject of contention on many online forums, but personally, it even made stop trying out Nix in the beginning and to give up on more certain parts of it a couple of times. I found the tutorials, posts, manuals, etc. confusing because sometimes there were sudden context switches, and not knowing what other aspects of NIx there are, I couldn't connect the dots (heh, still can't most of the time).
[2]: Thesis, page 21 (PDF page 29), 2nd paragraph from bottom.
[3]: My first try was "A (new) paradigm of building digital artifacts3 (where the input can be anything that require assembly), and managing the build results., but the term "digital artifact" seems just as overloaded as "component" or "package":