Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document Store Derivations and Deriving Paths #12290

Merged
merged 32 commits into from
Feb 10, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
a654cc2
WIP document derivations and deriving paths
Ericson2314 Nov 16, 2023
5eeae83
WIP
Ericson2314 Nov 18, 2024
1926161
Small tweak
Ericson2314 Nov 18, 2024
a8b1e3f
WIP moving info from `builtins.derivation` docs
Ericson2314 Nov 18, 2024
16bd5a4
Merge branch 'master' into document-derivation-and-deriving-path
Ericson2314 Jan 19, 2025
97977b7
Fix formatting issue
Ericson2314 Jan 19, 2025
4b29ca8
Fix typo
Ericson2314 Jan 19, 2025
4b9fb54
Derivation doc improvements
Ericson2314 Jan 19, 2025
87c559e
More doc improvements
Ericson2314 Jan 19, 2025
96a46f9
Move builder execution to store layer
Ericson2314 Jan 19, 2025
6246901
Formatting example
Ericson2314 Jan 19, 2025
80e6942
Fix redirect
Ericson2314 Jan 19, 2025
e77338d
Fix redirects more
Ericson2314 Jan 19, 2025
fb7fc6a
Fix another URL in the docs
Ericson2314 Jan 19, 2025
0cd1e78
Change deriving path pseudo-code
Ericson2314 Jan 19, 2025
05a38b8
Further reworking
Ericson2314 Jan 20, 2025
e409ea7
Apply suggestions from code review
Ericson2314 Jan 20, 2025
3d3a86d
Merge branch 'master' into document-derivation-and-deriving-path
Ericson2314 Jan 20, 2025
f3fa9cf
Clean out some TODOs
Ericson2314 Jan 20, 2025
f2d5be9
Remove store-layer `__structuredAttrs` docs
Ericson2314 Jan 20, 2025
86f23c2
Fix build failure
Ericson2314 Jan 20, 2025
3a3f6ad
Rework "Processing outputs" slightly
Ericson2314 Jan 20, 2025
b027e22
Fix missing link
Ericson2314 Jan 20, 2025
f003b8c
Merge branch 'master' into document-derivation-and-deriving-path
Ericson2314 Feb 6, 2025
54d684e
Call it "store derivation", again, but keep "derivation expression"
Ericson2314 Feb 6, 2025
094d352
Apply suggestions from code review
Ericson2314 Feb 9, 2025
298d199
Apply suggestions from code review
Ericson2314 Feb 10, 2025
1e2dfce
Apply suggestions from code review
Ericson2314 Feb 10, 2025
ea21555
More RFC 92
Ericson2314 Feb 10, 2025
9847acc
Move up `system`
Ericson2314 Feb 10, 2025
cba35b1
Update doc/manual/source/store/drv.md
Ericson2314 Feb 10, 2025
c343e05
Update src/libexpr/primops.cc
roberth Feb 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/manual/redirects.js
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,9 @@ const redirects = {
"scoping-rules": "scoping.html",
"string-literal": "string-literals.html",
},
"language/derivations.md": {
"builder-execution": "store/drv/building.md#builder-execution",
},
"installation/installing-binary.html": {
"linux": "uninstall.html#linux",
"macos": "uninstall.html#macos",
Expand Down
2 changes: 2 additions & 0 deletions doc/manual/source/SUMMARY.md.in
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
- [Store Object](store/store-object.md)
- [Content-Addressing Store Objects](store/store-object/content-address.md)
- [Store Path](store/store-path.md)
- [Store Derivation and Deriving Path](store/drv.md)
- [Building](store/building.md)
- [Store Types](store/types/index.md)
{{#include ./store/types/SUMMARY.md}}
- [Nix Language](language/index.md)
Expand Down
2 changes: 1 addition & 1 deletion doc/manual/source/architecture/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ It can also execute build plans to produce new data, which are made available to
A build plan itself is a series of *build tasks*, together with their build inputs.

> **Important**
> A build task in Nix is called [derivation](@docroot@/glossary.md#gloss-derivation).
> A build task in Nix is called [store derivation](@docroot@/glossary.md#gloss-store-derivation).

Each build task has a special build input executed as *build instructions* in order to perform the build.
The result of a build task can be input to another build task.
Expand Down
12 changes: 7 additions & 5 deletions doc/manual/source/command-ref/nix-env/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@ It is based on the current generation of the active [profile](@docroot@/command-

The arguments *args* map to store paths in a number of possible ways:

- By default, *args* is a set of [derivation] names denoting derivations in the [default Nix expression].
- By default, *args* is a set of names denoting derivations in the [default Nix expression].
These are [realised], and the resulting output paths are installed.
Currently installed derivations with a name equal to the name of a derivation being added are removed unless the option `--preserve-installed` is specified.

[derivation]: @docroot@/glossary.md#gloss-derivation
[derivation expression]: @docroot@/glossary.md#gloss-derivation-expression
[default Nix expression]: @docroot@/command-ref/files/default-nix-expression.md
[realised]: @docroot@/glossary.md#gloss-realise

Expand Down Expand Up @@ -66,11 +66,11 @@ The arguments *args* map to store paths in a number of possible ways:
This can be used to override the priority of the derivations being installed.
This is useful if *args* are [store paths], which don't have any priority information.

- If *args* are [store derivations](@docroot@/glossary.md#gloss-store-derivation), then these are [realised], and the resulting output paths are installed.
- If *args* are [store paths] that point to [store derivations][store derivation], then those store derivations are [realised], and the resulting output paths are installed.

- If *args* are [store paths] that are not store derivations, then these are [realised] and installed.
- If *args* are [store paths] that do not point to store derivations, then these are [realised] and installed.

- By default all [outputs](@docroot@/language/derivations.md#attr-outputs) are installed for each [derivation].
- By default all [outputs](@docroot@/language/derivations.md#attr-outputs) are installed for each [store derivation].
This can be overridden by adding a `meta.outputsToInstall` attribute on the derivation listing a subset of the output names.

Example:
Expand Down Expand Up @@ -122,6 +122,8 @@ The arguments *args* map to store paths in a number of possible ways:
manifest.nix
```

[store derivation]: @docroot@/glossary.md#gloss-store-derivation

# Options

- `--prebuilt-only` / `-b`
Expand Down
5 changes: 4 additions & 1 deletion doc/manual/source/command-ref/nix-env/query.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,10 @@ derivation is shown unless `--no-name` is specified.

- `--drv-path`

Print the path of the [store derivation](@docroot@/glossary.md#gloss-store-derivation).
Print the [store path] to the [store derivation].

[store path]: @docroot@/glossary.md#gloss-store-path
[store derivation]: @docroot@/glossary.md#gloss-derivation

- `--out-path`

Expand Down
4 changes: 2 additions & 2 deletions doc/manual/source/command-ref/nix-instantiate.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ standard input.
- `--eval`

Just parse and evaluate the input files, and print the resulting
values on standard output. No instantiation of store derivations
takes place.
values on standard output.
Store derivations are not serialized and written to the store, but instead just hashed and discarded.

> **Warning**
>
Expand Down
53 changes: 27 additions & 26 deletions doc/manual/source/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,31 +19,35 @@

Besides content addressing, the Nix store also uses [input addressing](#gloss-input-addressed-store-object).

- [derivation]{#gloss-derivation}
- [store derivation]{#gloss-store-derivation}

A description of a build task. The result of a derivation is a
store object. Derivations declared in Nix expressions are specified
using the [`derivation` primitive](./language/derivations.md). These are
translated into low-level *store derivations* (implicitly by
`nix-build`, or explicitly by `nix-instantiate`).
A single build task.
See [Store Derivation](@docroot@/store/drv.md#store-derivation) for details.

[derivation]: #gloss-derivation
[store derivation]: #gloss-store-derivation

- [store derivation]{#gloss-store-derivation}
- [derivation path]{#gloss-derivation-path}

A [derivation] represented as a `.drv` file in the [store].
It has a [store path], like any [store object].
It is the [instantiated][instantiate] form of a derivation.
A [store path] which uniquely identifies a [store derivation].

Example: `/nix/store/g946hcz4c8mdvq2g8vxx42z51qb71rvp-git-2.38.1.drv`
See [Referencing Store Derivations](@docroot@/store/drv.md#derivation-path) for details.

See [`nix derivation show`](./command-ref/new-cli/nix3-derivation-show.md) (experimental) for displaying the contents of store derivations.
Not to be confused with [deriving path].

[store derivation]: #gloss-store-derivation
[derivation path]: #gloss-derivation-path

- [derivation expression]{#gloss-derivation-expression}

A description of a [store derivation] in the Nix language.
The output(s) of a derivation are store objects.
Derivations are typically specified in Nix expressions using the [`derivation` primitive](./language/derivations.md).
These are translated into store layer *derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`).

[derivation expression]: #gloss-derivation-expression

- [instantiate]{#gloss-instantiate}, instantiation

Save an evaluated [derivation] as a [store derivation] in the Nix [store].
Translate a [derivation expression] into a [store derivation].

See [`nix-instantiate`](./command-ref/nix-instantiate.md), which produces a store derivation from a Nix expression that evaluates to a derivation.

Expand All @@ -55,7 +59,7 @@

This can be achieved by:
- Fetching a pre-built [store object] from a [substituter]
- Running the [`builder`](@docroot@/language/derivations.md#attr-builder) executable as specified in the corresponding [derivation]
- Running the [`builder`](@docroot@/language/derivations.md#attr-builder) executable as specified in the corresponding [store derivation]
- Delegating to a [remote machine](@docroot@/command-ref/conf-file.md#conf-builders) and retrieving the outputs
<!-- TODO: link [running] to build process page, #8888 -->

Expand All @@ -73,7 +77,7 @@

- [fixed-output derivation]{#gloss-fixed-output-derivation} (FOD)

A [derivation] where a cryptographic hash of the [output] is determined in advance using the [`outputHash`](./language/advanced-attributes.md#adv-attr-outputHash) attribute, and where the [`builder`](@docroot@/language/derivations.md#attr-builder) executable has access to the network.
A [store derivation] where a cryptographic hash of the [output] is determined in advance using the [`outputHash`](./language/advanced-attributes.md#adv-attr-outputHash) attribute, and where the [`builder`](@docroot@/language/derivations.md#attr-builder) executable has access to the network.

- [store]{#gloss-store}

Expand Down Expand Up @@ -188,7 +192,7 @@
>
> The contents of a `.nix` file form a Nix expression.

Nix expressions specify [derivations][derivation], which are [instantiated][instantiate] into the Nix store as [store derivations][store derivation].
Nix expressions specify [derivation expressions][derivation expression], which are [instantiated][instantiate] into the Nix store as [store derivations][store derivation].
These derivations can then be [realised][realise] to produce [outputs][output].

> **Example**
Expand Down Expand Up @@ -230,14 +234,14 @@

- [output]{#gloss-output}

A [store object] produced by a [derivation].
A [store object] produced by a [store derivation].
See [the `outputs` argument to the `derivation` function](@docroot@/language/derivations.md#attr-outputs) for details.

[output]: #gloss-output

- [output path]{#gloss-output-path}

The [store path] to the [output] of a [derivation].
The [store path] to the [output] of a [store derivation].

[output path]: #gloss-output-path

Expand All @@ -246,14 +250,11 @@

- [deriving path]{#gloss-deriving-path}

Deriving paths are a way to refer to [store objects][store object] that ar not yet [realised][realise].
This is necessary because, in general and particularly for [content-addressed derivations][content-addressed derivation], the [output path] of an [output] is not known in advance.
There are two forms:
Deriving paths are a way to refer to [store objects][store object] that might not yet be [realised][realise].

- *constant*: just a [store path]
It can be made [valid][validity] by copying it into the store: from the evaluator, command line interface or another store.
See [Deriving Path](./store/drv.md#deriving-path) for details.

- *output*: a pair of a [store path] to a [derivation] and an [output] name.
Not to be confused with [derivation path].

- [deriver]{#gloss-deriver}

Expand Down
96 changes: 9 additions & 87 deletions doc/manual/source/language/derivations.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Derivations

The most important built-in function is `derivation`, which is used to describe a single derivation:
a specification for running an executable on precisely defined input files to repeatably produce output files at uniquely determined file system paths.
The most important built-in function is `derivation`, which is used to describe a single store-layer [store derivation].
Consult the [store chapter](@docroot@/store/drv.md) for what a store derivation is;
this section just concerns how to create one from the Nix language.

It takes as input an attribute set, the attributes of which specify the inputs to the process.
This builtin function takes as input an attribute set, the attributes of which specify the inputs to the process.
It outputs an attribute set, and produces a [store derivation] as a side effect of evaluation.

[store derivation]: @docroot@/glossary.md#gloss-store-derivation
Expand All @@ -15,7 +16,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
- [`name`]{#attr-name} ([String](@docroot@/language/types.md#type-string))

A symbolic name for the derivation.
It is added to the [store path] of the corresponding [store derivation] as well as to its [output paths](@docroot@/glossary.md#gloss-output-path).
See [derivation outputs](@docroot@/store/drv.md#outputs) for what this is affects.

[store path]: @docroot@/store/store-path.md

Expand All @@ -28,17 +29,12 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
> }
> ```
>
> The store derivation's path will be `/nix/store/<hash>-hello.drv`.
> The derivation's path will be `/nix/store/<hash>-hello.drv`.
> The [output](#attr-outputs) paths will be of the form `/nix/store/<hash>-hello[-<output>]`

- [`system`]{#attr-system} ([String](@docroot@/language/types.md#type-string))

The system type on which the [`builder`](#attr-builder) executable is meant to be run.

A necessary condition for Nix to build derivations locally is that the `system` attribute matches the current [`system` configuration option].
It can automatically [build on other platforms](@docroot@/language/derivations.md#attr-builder) by forwarding build requests to other machines.

[`system` configuration option]: @docroot@/command-ref/conf-file.md#conf-system
See [system](@docroot@/store/drv.md#system).

> **Example**
>
Expand Down Expand Up @@ -68,7 +64,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect

- [`builder`]{#attr-builder} ([Path](@docroot@/language/types.md#type-path) | [String](@docroot@/language/types.md#type-string))

Path to an executable that will perform the build.
See [builder](@docroot@/store/drv.md#builder).

> **Example**
>
Expand Down Expand Up @@ -117,7 +113,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect

Default: `[ ]`

Command-line arguments to be passed to the [`builder`](#attr-builder) executable.
See [args](@docroot@/store/drv.md#args).

> **Example**
>
Expand Down Expand Up @@ -239,77 +235,3 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
passed as an empty string.

<!-- FIXME: add a section on output attributes -->

## Builder execution

The [`builder`](#attr-builder) is executed as follows:

- A temporary directory is created under the directory specified by
`TMPDIR` (default `/tmp`) where the build will take place. The
current directory is changed to this directory.

- The environment is cleared and set to the derivation attributes, as
specified above.

- In addition, the following variables are set:

- `NIX_BUILD_TOP` contains the path of the temporary directory for
this build.

- Also, `TMPDIR`, `TEMPDIR`, `TMP`, `TEMP` are set to point to the
temporary directory. This is to prevent the builder from
accidentally writing temporary files anywhere else. Doing so
might cause interference by other processes.

- `PATH` is set to `/path-not-set` to prevent shells from
initialising it to their built-in default value.

- `HOME` is set to `/homeless-shelter` to prevent programs from
using `/etc/passwd` or the like to find the user's home
directory, which could cause impurity. Usually, when `HOME` is
set, it is used as the location of the home directory, even if
it points to a non-existent path.

- `NIX_STORE` is set to the path of the top-level Nix store
directory (typically, `/nix/store`).

- `NIX_ATTRS_JSON_FILE` & `NIX_ATTRS_SH_FILE` if `__structuredAttrs`
is set to `true` for the derivation. A detailed explanation of this
behavior can be found in the
[section about structured attrs](./advanced-attributes.md#adv-attr-structuredAttrs).

- For each output declared in `outputs`, the corresponding
environment variable is set to point to the intended path in the
Nix store for that output. Each output path is a concatenation
of the cryptographic hash of all build inputs, the `name`
attribute and the output name. (The output name is omitted if
it’s `out`.)

- If an output path already exists, it is removed. Also, locks are
acquired to prevent multiple Nix instances from performing the same
build at the same time.

- A log of the combined standard output and error is written to
`/nix/var/log/nix`.

- The builder is executed with the arguments specified by the
attribute `args`. If it exits with exit code 0, it is considered to
have succeeded.

- The temporary directory is removed (unless the `-K` option was
specified).

- If the build was successful, Nix scans each output path for
references to input paths by looking for the hash parts of the input
paths. Since these are potential runtime dependencies, Nix registers
them as dependencies of the output paths.

- After the build, Nix sets the last-modified timestamp on all files
in the build result to 1 (00:00:01 1/1/1970 UTC), sets the group to
the default group, and sets the mode of the file to 0444 or 0555
(i.e., read-only, with execute permission enabled if the file was
originally executable). Note that possible `setuid` and `setgid`
bits are cleared. Setuid and setgid programs are not currently
supported by Nix. This is because the Nix archives used in
deployment have no concept of ownership information, and because it
makes the build result dependent on the user performing the build.
5 changes: 3 additions & 2 deletions doc/manual/source/language/import-from-derivation.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,9 @@ Boxes are data structures, arrow labels are transformations.
| evaluate | | |
| | | | |
| V | | |
| .------------. | | .------------------. |
| | derivation |----|-instantiate-|->| store derivation | |
| .------------. | | |
| | derivation | | | .------------------. |
| | expression |----|-instantiate-|->| store derivation | |
| '------------' | | '------------------' |
| | | | |
| | | realise |
Expand Down
6 changes: 3 additions & 3 deletions doc/manual/source/language/string-interpolation.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ Rather than writing
"--with-freetype2-library=" + freetype + "/lib"
```

(where `freetype` is a [derivation]), you can instead write
(where `freetype` is a [derivation expression]), you can instead write

[derivation]: @docroot@/glossary.md#gloss-derivation
[derivation expression]: @docroot@/glossary.md#gloss-derivation-expression

```nix
"--with-freetype2-library=${freetype}/lib"
Expand Down Expand Up @@ -148,7 +148,7 @@ An expression that is interpolated must evaluate to one of the following:
- `__toString` must be a function that takes the attribute set itself and returns a string
- `outPath` must be a string

This includes [derivations](./derivations.md) or [flake inputs](@docroot@/command-ref/new-cli/nix3-flake.md#flake-inputs) (experimental).
This includes [derivation expressions](./derivations.md) or [flake inputs](@docroot@/command-ref/new-cli/nix3-flake.md#flake-inputs) (experimental).

A string interpolates to itself.

Expand Down
Loading
Loading