CLI stabilization effort

[tomberek]

This is a DRAFT
Tracking for the stabilizing effort:

Tasks

1. consider feature mappings with legacy CLI (not expected to be 1-1)
2. review correctness of command
3. do all --help options make sense
4. create any issues or PRs to resolve the above, we will link it here for tracking
5. verify "ready for stabilization"

The goal isn't to stabilize the whole CLI at once. A bunch of less-frequent commands can wait for a bit while we focus on the most important ones.

First batch of stabilization

• nix daemon
  • Port the flags of nix-daemon to nix daemon #8788
  • Test the nix daemon command #9647
• nix nar
  • Stabilise nix nar and subcommands #8875
  • request for contributions: make this usable as a general utility. eg: the unpack/extract
  • rename to pack/unpack
  • (non-blockers) refine other subcommands such as cat, ls, tar conversions
• nix hash
  • rename sub-commands: Stabilise nix hash and subcommands #8876
  • nix hash convert [--type sha256] --from (base16|...|sri) --to base64 strings...
    • Add nix hash convert #9452
  • nix hash path
    • nix hash path, text hashing for nix store add, and preparatory refactors #9815
• nix key
  • nix key: Add a newline to the output #8849
  • Stabilise nix key #8877
• new-command: nix config
  • Check
  • Show
• nix store
  • nix store gc nix store gc: Stabilize #7608
    • Make 'nix store gc' use the auto-GC policy  #7851 : honor min-free and max-free config options, and remove --max flag
      • note: --max is closer to the understanding of --min and can/should also be idempotent relative to disk space?
  • nix store ping Stabilise nix store ping #8914
    • trusted returned as integer is awkward, should be a boolean
    • rename to nix store info
  • nix store add-file,nix store add-path Stabilise nix store add #8915
    • --recursive should be the default due to history and to match string-interpolation behavior
    • need clarification of how to specify outputHashMode, outputHashAlgo, etc.
    • expose as --key value or --value1|--value2 create PR to CLI guidelines
  • nix store import, nix store export Not a blocker for stabilisation
  • nix store delete
    • provide better information about why something was not deleted Not required for stabilisation
      • example: Cannot delete store path with no roots #6141
    • Make nix store delete ./result work #9475 Not required for stabilisation
  • nix store optimise
    • stabilise as is!

To be discussed

Stuff that hasn't been processed yet

• nix copy

  • Stabilize nix copy #8011

• nix describe-stores nix help-stores

  • nix describe-stores --json format is not extensible #7824
  • Move nix {show-config,doctor} under subcommands #7672

• nix path-info

  • Categorize path-info and C++ path info fields #10498
  • Rework the store-side interfaceo

• nix build

  • https://discourse.nixos.org/t/2024-02-26-nix-team-meeting-minutes-128/40496#nix-build-7

• nix develop

• nix print-dev-env

• nix shell

• nix run

• nix search

• nix repl

  • Technically already stable

• nix eval

• nix bundle

• nix edit

• nix fmt

• nix log

• nix registry

• nix why-depends

• nix derivation

  • maybe Derivation JSON env does not adhere to the JSON guidelines #9866

• nix profile: nix profile has bad UX #7966

• nix realisation

• nix upgrade-nix

• nix flake

Global concerns

• settings
  • Triage and break up the global settings #5638
  • Move settings from Settings to LocalStoreConfig #7740
  • Hide experimental settings #7609
• config
• flags
  • Remove --print-build-logs from various commands #7820
  • Stuctured command stability #7611
• layering, see: [RFC 0136] A plan to stabilize the new CLI and Flakes incrementally rfcs#136
• (non-blocking) Can't add a file from stdin #7773

Blocked on installables

• nix shell
  • Rename 'nix shell' #4715
  • nix shell does not insert PATH items in command line order #7905
  • ...
• ...

To stabilise in a later batch

These are low priority and have more involved background issues we have to clarify first

• nix store cat
• nix store ls
• nix store copy-log
• nix store copy-sigs
• nix store diff-closures
• nix store make-content-addressed
• nix store path-from-hash-part
• nix store prefetch-file
• nix store repair
• nix store verify
  • document the defaults for the options
  • ensure that sigs-needed only takes into account the cryptographic portion of the key, not the name
• nix store sign
  • see previous: Unsigned derivations uploaded to S3 #6960 (comment)
  • QUESTION: sign only until other signatures encountered (limited -r)
  • --key-file - to read key from stdin, exclusive with --stdin, but then how to pass both via pipe?
  • QUESTION: add a --pipe FD for which --stdin is alias for --pipe /dev/fd/0?

References

CLI Guidelines
legacy CLI mapping

[Ericson2314]

#7740 / #5638 being hard I think it something to think about prior to command stabilization.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-01-27-nix-team-meeting-minutes-27/25434/1

[fricklerhandwerk]

Discussed in the Nix team meeting 2023-02-10:

Decision: work through one complete command at a time and write down the process. once finished and everyone is confident to reproduce that process on one's own, do the rest of the work in parallel.

Complete discussion

• @tomberek will clarify the exact things to do
  • @Ericson2314: the idea was to take one command at a time to figure out these things to begin with
  • @fricklerhandwerk: why don't we do it together for one command?
  • @Ericson2314: we already started with nix store gc
  • @fricklerhandwerk: let's take one hour together in the work meetings to attack this and write down what we do. when one command is finished (it is likely to take months at that pace), and everyone is confident to reproduce the process on one's own given the documentation we write, we can parallelise it for all the other commands
    • agreement
    • @roberth: several months are a blip in relation to how long it took to develop the new CLI anyway
  • @thufschmitt: will make a PR to put on the agenda for next time
    • won't be there myself though

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-02-10-nix-team-meeting-minutes-31/25438/1

[Ericson2314]

Two topics (with my PR fixes, if we like my solution) that I think would be good to track:

• Gating experimental settings on experimental features --- Hide experimental settings #7609
• Hiding freeform prose on experimental topics in --help --- Stuctured command stability #7611

(My middle PR, #7610, corresponds to #7820 already listed here, so there is no need to link the issue directly.)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-02-10-nix-team-meeting-minutes-32/25442/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-44/25546/1

[fricklerhandwerk]

Discussed general design considerations in Nix team meeting 2023-03-20:

@Ericson2314 and @fricklerhandwerk prefer to have a clean separation of composable "plumbing" commands (reflecting architectural separation of store and evaluation layers) and convenient "porcelain" commands that aim for brevity and common use cases. Specifically it should be nix store copy[-(closure|path|...)] <store path> and nix copy <installable>. @edolstra noted that the original goal of the CLI redesign was not to have that distinction, and that this has been decided long ago. The merit of this was disputed. We also briefly touched upon perceived problems with nix-env, where one can observe a similar tension.

There was no conclusion.

Although apparently this has been talked about multiple times in the past years, and @edolstra is (rightly) annoyed about rehashing the arguments, there seems to be no record of the prior discussions. (Help digging them out highly appreciated!)

Complete discussion

• @thufschmitt: last time we decided to stabilise a subset of nix copy to bring back nix-push functionality
  • possibly adding a nix-push wrapper just to have the interface in place
• @fricklerhandwerk: nix copy seems to only operate at the store level, why is it not nix store copy?
  • @thufschmitt: one reason is that it's very important
  • @edolstra: the question is how often one uses this command
    • don't think it's that common
  • @thufschmitt: nix store copy sounds like you'd copy a store
  • @Ericson2314: we have nix store copy-log
    • @thufschmitt: we could have nix store copy-path
• @Ericson2314: nix copy is hard to stabilise in general
  • we'd probably need a stable store (to/from) syntax first
  • we also haven't looked over the store options that much yet
• @thufschmitt: two approaches
  1. re-write nix-push in terms of old CLI idioms
  2. stabilise a smallest possible subset of nix copy
     • challenge is how to make that subset not awkward
• @Ericson2314: we could have a nix store copy command that only works on paths
  • it would be good to have commands that do one thing well
  • @edolstra: disagree. we had that in the old CLI, while that was architecturally nice, it's not good for the user
    • if users want to operate on store paths, they should pass store paths
    • I want to be able to do nix copy nixpkgs#hello
    • @Ericson2314: just as we'd have a top-level nix gc we could also have that
      • want to separate porcelain from plumbing
      • @edolstra: the porcelain here is in the installables
        • @Ericson2314: that is actually the problem. there is an incongruency in behavior for different types of installables
          • for CA derivations, if you specify something higher-level where it has to figure out "realizations", it will do different things than just for store paths
      • @fricklerhandwerk: I see the problem that installables are polymorphic, and it's not clear for beginners at all what the common properties are
        • @Ericson2314: polymorphism is probably okay, as long as the downcast is unconditional for all of them and it always has the same result
          • @fricklerhandwerk: passing -E '1+1' as an installable will throw an error that doesn't help anywone who doesn't know how the underlying mechanism works
            • why you should know what a derivation even is, just because it's possible to pass something to nix copy that is not in general copyable?
          • @edolstra: this ship has sailed a long time ago
            • the new CLI is all about not having separate commands for the store layer and the expression layer
• (some discussion about CLI design and how the old and the new relate)
  • @fricklerhandwerk: agree with @Ericson2314 that we should keep an eye on both highly composable plumbing commands and also end-user facing convenience commands (which may be less conceptually sharp)
    • @edolstra: this discussion is 5 years late, because it is the underlying paradigm of the new CLI not to make that distinction explicit
  • @roberth: related nix copy --to doesn't copy and attempts a build #7095
    • @Ericson2314: yes, why would nix copy run a build?
    • @edolstra: if you want to copy a path, the postcondition should be that the path is there
• (some more debate around how nix-env apparently creates confusion, which is related to the interface design issue)
• no conclusion, but apparently we may have to agree on priorities

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-04-14-nix-team-meeting-minutes-48/27358/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-team-report-2022-10-2023-03/27486/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-08-28-nix-team-meeting-minutes-83/32290/1

[roberth]

Meeting notes 2023-09-01

nix store gc

What about the gc-related queries?

• Maybe add --dry-run and --json to nix store gc?
• or to nix store delete
• How do we structure the "reverse" queries? Just gc roots? or more?
• Returning only some things can be confusing
• Returning everything may be slow
• @roberth: If we return to little, then the users has to backtrack with docs, and then still run the slow thing, worst of both worlds

nix store ping

What does --json return?
Why does trusted return an int? Change to bool-or-null

TODO copy note from comment on issue

Rename to nix store info.

nix store add-file

add-path?

@edolstra: benefit of explicit add-file is that it matches fetchurl
@Ericson2314: a single command with multiple modes, flags seems more sensible
@thufschmitt: what's the use case?
@edolstra: generally prefetching
@roberth: also RFC 92 dynamic derivations?
@thufschmitt: seem similar
@ericsion2314: the varying hashing methods have always been a source of confusion to me. Making this explicit is helpful for learning
@edolstra: names?
@roberth: match the code?
Too long and inconvenient
@roberth: short flags that are neatly categorized in the docs?
@edolstra: should match derivation attr recursive, flat
@roberth: recursive should be the default?
Make it match the language? Then it should be recursive or flat
@roberth: recursive-or-flat as a mode? It names the language behavior wrong! The language uses recursive in its string coercions.
Should it match outputHashAlgo / outputHashMethod?
@thufschmitt: It would be great for the language and CLI to agree
@Ericson2314: I think it is very good if the CLI is "subliminally" teaching people the most compressed/efficient mental model by never missing and opportunity to connect things that are the same. (The less users need to consult the docs the better.) So if recursive hashing is deeply related to NAR hashing, they should share a name. (I took a lot of time to "academically" understand Nix and refine my mental model, but most users won't take the time to "relearn" that way. So we want to give them the right mental model from the beginning.)

@thufschmitt: Do we want key-value options or flags? (--method=foo or --foo)
@edolstra: Prefer brief options. Would like to avoid string-value for enums.
@roberth: This command is for scripts and power users, so brevity is not a priority.
@tomberek: What should be the CLI guideline?
@edolstra: enum key-value options can always be turned into single flags
@Ericson2314: such flags are naturally booleans, whereas we're dealing with an enum
@roberth: key-value matches the language, matches derivations

@edolstra: We have precendent for flattening enum-valued options to mutually exclusive flags, e.g. sandbox
@roberth: for tab completion we want a consistent prefix
@edolstra: Implementation effort may be higher for enum case
@roberth: Should be close: either you specify more flags, or you specify a string mapping
@Ericson2314: Would be happy to implement infra for enum options if still needed.
@ericson3214: Support text for manually adding derivations in aterm format (e.g. for RFC 92, or just for complete coverage)

@tomberek: Let's also update / clarify the CLI guidelines.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-01-nix-team-meeting-minutes-84/32466/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixcon-governance-workshop/32705/9

[anka-213]

• Maybe add --dry-run and --json to nix store gc?

I would argue that almost all of the commands in the new CLI need a --dry-run flag (#7227)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-25-nix-team-meeting-minutes-89/33489/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-10-09-nix-team-meeting-minutes-93/34388/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/stabilising-the-new-nix-command-line-interface/35531/1

[roberth]

Just had a conversation on mastodon, where nix store gc was compared to the legacy command with -d (after some confusion).
The lack of generation deletion is somewhat of a regression from the user perspective, so we can only really claim parity when we have a nix profile subcommand that does root deletion and gc.
Would nix profile collect be a good name? My reasoning is that it doesn't collect just the garbage.

[fricklerhandwerk]

nix profile clean[up]

What is it supposed to collect?

Although in any case it can be slightly misleading because if that command does the same as nix-collect-garbage -d, users may be unpleasantly surprised that other things not rooted in stale profiles also get collected. So maybe, in fact, that's what the command should do?

[thufschmitt]

@roberth : I think it would make sense to have both (a nix store gc that doesn't care about the profiles and nix profile collect/cleanup that also removes the old profiles (maybe in a more targeted way building on top of #7239?).

[fricklerhandwerk]

maybe in a more targeted way building on top of #7239

Exactly! We should address the mechanism first before piling up policy.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-52/38343/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/contribute-to-the-nix-2023-recap/37484/13

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-team-member-suggests-removing-flakes-data-suggest-it-isnt-a-good-idea-please-remove-the-experimental-flag-instead/54959/65

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-team-member-suggests-removing-flakes-data-suggest-it-isnt-a-good-idea-please-remove-the-experimental-flag-instead/54959/151