Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite it in python, rust, or go #789

Open
iFreilicht opened this issue Sep 20, 2024 · 36 comments
Open

Rewrite it in python, rust, or go #789

iFreilicht opened this issue Sep 20, 2024 · 36 comments
Labels
enhancement New feature or request

Comments

@iFreilicht
Copy link
Contributor

iFreilicht commented Sep 20, 2024

Problem

Disko is currently somewhat tricky to maintain. Part of this comes from the current architecture that puts nix itself at the center of the entire operation, and uses it as a crude templating engine for bash scripts.

This makes implementation of some features difficult. A few examples (I'll add more when I come across them):

Proposed solution

Switch to a different architecture where the nix files only define configuration values, and a bigger program written in a proper language (not bash or perl) runs nix eval --json and works with the JSON output to plan and execute all the required actions.

Considerations

Compatibility

The interface for users should not change at all. disko.devices should have the exact same type and result in the same configuration values as before.

Some programs (most notable nixos-anywhere, AFAIK) use disko's diskoScript output and call this script directly. Because this is a derivation, it should be possible to just swap them out no matter the language, as long as the resulting path has a binary with the same name. However, a simple shim with a single-line script that just calls the CLI with the required flags might be sufficient for this.

Language to use

The language should make it easy for others to contribute, have decent error-handling capabilities and be well-suited to write a CLI in.

I am therefore immediately discarding bash (or any other shell language), perl, JS, TS and Haskell.

I see three main contenders: python, rust, and go. They are all well-known and have good general support. Rust is additionally loved in the nix community for its ethos of correctness, python is super easy to write, and go is kind-of a middle-ground between the two.

One thing to consider with this choice might be closure size. As disko is often run in installers, we can expect most dependencies to not be present yet (this is the singular advantage of bash; it is guaranteed to already be present). The closure sizes of the main executable vary quite a bit:

$ nix path-info --closure-size -h nixpkgs#cargo^out
/nix/store/j6vpbh78ngj2x7jrg27pqjn2cd5mzywl-cargo-1.79.0          1.2G
$ nix path-info --closure-size -h nixpkgs#go^out                                                                                                                                                                                                                                                                                          
/nix/store/kqf4gwhq1yrchvmvqjh4adc4y9hxp9xk-go-1.22.5    248.0M
$ nix path-info --closure-size -h nixpkgs#python3^out                                                                                                                                                                                                                                                                                    
/nix/store/l014xp1qxdl6gim3zc0jv3mpxhbp346s-python3-3.12.4       166.1M

It seems somewhat unreasonable to me to require the user to download 1.2 gigabytes just to be able to run disko. A binary cache could alleviate this, though this has additional usability implications.

In the case of rust and go, there would also be compilation times added, though it's hard to say if this will matter much given the size of this project.

In general, I don't yet have a strong preference for which language to use.

@Mic92
Copy link
Member

Mic92 commented Sep 21, 2024

With nixos-anywhere we need to be able to build disko from source in a memory constraint environment. Currently our kexec image targets 1GB of RAM memory usage. We cannot depend on a binary cache just now because disko is not part of nixpkgs and the disko configuration is derived from the module system. We do have a dependency of python in the installer. We almost got rid of perl because of the perlless profile except for nixos-generate-config.

As see just now is that both the go and rust compiler might be almost as large as our kexec image (also this is compressed).

@Mic92
Copy link
Member

Mic92 commented Sep 21, 2024

One nice property of bash besides the really bad error handling is that its execution is very easy to introspect with set -x. So I think whatever replaces the bash code should be also be easy to debug at runtime.

@Lassulus
Copy link
Collaborator

also a problem of the rewrite is, that we can only rewrite the creation and mounting, the nixos configuration has still to be generated in the module system, so we would have two parallel implementations of the disko code.

@Mic92
Copy link
Member

Mic92 commented Sep 21, 2024

I don't follow this logic. We already have _create and _mount in bash so there is already a second imperative implementation next to the NixOS configuration.

@Lassulus
Copy link
Collaborator

yes, but instead of templating bash we would be templating rust, python or go code?

@iFreilicht
Copy link
Contributor Author

@Mic92 that's very useful information, thank you! That makes it seem like python is the only viable option.

We do have a dependency of python in the installer.

Which installer? I booted a NixOS ISO and didn't see python in there, but I assume you're talking about something else?

also a problem of the rewrite is, that we can only rewrite the creation and mounting, the nixos configuration has still to be generated in the module system, so we would have two parallel implementations of the disko code.

This is not a problem, it's just a different way of modelling the problem space. Of course, we have to avoid duplicate implementations. If a value is required by both the nixos configuration and the mount/format code, its value must be computed in nix code. But there's no need for all the logic that actually creates partitions and mounts them to be implemented as templated bash, we can choose to implement them however we want.

yes, but instead of templating bash we would be templating rust, python or go code?

No, not at all. My goal is is remove all _create and _mount attributes. Python code is awfully tricky to template because it's indentation dependent. Go or Rust might be simpler in that regard, but it's still a bad solution. Instead, the create/mount code would shell out to nix, evaluate the configuration as a JSON and then use that as an input.

It's true that we would have to change the structure a little, so next to zfs_fs.nix, zfs.nix and zpool.nix there would be a zfs.py file, but I don't see how that's an issue.

One nice property of bash besides the really bad error handling is that its execution is very easy to introspect with set -x. So I think whatever replaces the bash code should be also be easy to debug at runtime.

Very true! Well wouldn't you know it, python allows you to do exactly that with the trace module.
Though I think it would also be fine to just do meticulous logging. In the past, I've just written a short wrapper around the exec primitive of the language to logged all inputs and outputs on any shell command invocation.

@Mic92
Copy link
Member

Mic92 commented Sep 21, 2024

Which installer? I booted a NixOS ISO and didn't see python in there, but I assume you're talking about something else?

Afaik it's used for systemd-boot installer, but it's not in $PATH - which might be a problem for disko actually. However we have it in our kexec image for sure for restoring network configuration.

yes, but instead of templating bash we would be templating rust, python or go code?

I wouldn't use templating at all. I would parse json instead and generate a plan based on that.
Json could be also used to inject package dependencies.

@Mic92
Copy link
Member

Mic92 commented Sep 21, 2024

However overall it might be quite ambiguous to re-write a 3k LOC project written in a terse language in one go. I would suggest if you want to go this route is to replace one smaller module with some prototype so we can see if this goes in the direction we are all happy with.

@sedlund
Copy link

sedlund commented Oct 3, 2024

nushell is an interesting option for this use case. it has much better error handling and debugging, and is meant to easily interop with existing shell tools that are heavily used in disko.

@iFreilicht
Copy link
Contributor Author

That's an interesting idea. It's been a few years since I had a look at nushell. I'll read up on it. I also considered xonsh, it has a similar advantage of allowing easy interop with shell tools while working with resulting values in the python world.

How well is nushell suited to building CLIs in your opinion?

@phaer
Copy link
Member

phaer commented Oct 3, 2024

How would nushell compare to pythonMinimal in terms of file size? The binary release sizes at https://github.com/nushell/nushell/releases seem to suggest they might be in the same ballpark? .

@Yethal
Copy link

Yethal commented Oct 3, 2024

+1 to nushell. Much better handling of structured data, easy interoperability and few if any footguns compared to bash
Plus it natively supports structured formats like json,yaml,toml etc so the 'generate json plan and feed it to external program' approach is feasible

@mmlb
Copy link

mmlb commented Oct 4, 2024

I'd throw elvish into the mix too. I haven't played around with nushell but I've started to convert all my scripts to elvish mainly because of the much better/enforced error handling. Its like what everyone expects from bash's set -euo pipefail but correct.

@Mic92
Copy link
Member

Mic92 commented Oct 5, 2024

We already depend on pythonMinimal for some filesystem utilities. Not just the networking script, that I added in nixos-images btw. Adding nushell would increase our image by 50MB (currently at 300MB), which seems like a lot.

@Yethal
Copy link

Yethal commented Oct 5, 2024

Yes but at the same time you could drop dependency on most of coreutils, jq and the like as all data processing can be done within Nushell directly.
Additionally Nu is a functional language at heart, has immutable variables by default, strict type system, scoped environments and strict parsing/evaluation separation so it's much closer in behavior to Nix itself than an imperative language like Python.
If you do decide to give Nushell a chance I'd be happy to assist with a PoC/code migration. While Nix itself is something I'm still learning Nushell has been my daily driver for the past three years and is the main tool used for any sort of automation at my day job

@iFreilicht
Copy link
Contributor Author

iFreilicht commented Oct 8, 2024

Yes but at the same time you could drop dependency on most of coreutils, jq and the like as all data processing can be done within Nushell directly.

Rewriting in python also allows dropping jq. Most other dependencies are always part of a nixos installer, so we likely don't save much there.

I also want to point out that many of the tools we use can't be dropped. For example, you might think that sys disks can be used to replace lsblk, but it doesn't give us enough information:

$ lsblk -fs
NAME                                            FSTYPE     FSVER LABEL     UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda1                                            zfs_member 5000  tank      15087946086092142717                                
└─sda                                                                                                                          
sdb1                                            ext4       1.0   scratch   89ae3545-751c-4954-a3fa-c6e4d1ccc1e0  215.6G     0% /mnt/scratch
└─sdb                                                                                                                          
sdc1                                            zfs_member 5000  tank      15087946086092142717                                
└─sdc                                                                                                                          
sdd1                                            ext4       1.0   nix_store 6404b8ba-ba70-4f3c-a6f5-460fd074fee3  172.8G    16% /nix/store
│                                                                                                                              /nix/store
└─sdd                                                                                                                          
sde1                                            zfs_member 5000  bigz      7887521794934826796                                 
└─sde                                                                                                                          
sde9                                                                                                                           
└─sde                                                                                                                          
sdf1                                            vfat       FAT32 BOOT      F488-86D9                               3.9G     3% /boot
└─sdf                                                                                                                          
sdf2                                            ext4       1.0   nixos     6ac40297-6c7a-4403-ad8e-dae1858e2344   21.5G     6% /var/lib/containers/storage/overlay
│                                                                                                                              /
└─sdf                                                                                                                          
dev-disk-byx2dpartlabel-diskx2dnix_storex2dswap swap       1               3806a8db-49d2-4aa0-97a3-8bd3f66f7029                [SWAP]
└─sdd2                                                                                                                         
  └─sdd                                                                                                                        
~> sys disks
╭────┬─────────────────────────────────────────────────┬──────┬─────────────────────┬───────────┬───────────┬───────────┬─────────╮
│  # │                     device                      │ type │        mount        │   total   │   free    │ removable │  kind   │
├────┼─────────────────────────────────────────────────┼──────┼─────────────────────┼───────────┼───────────┼───────────┼─────────┤
│  0 │ /dev/disk/by-partlabel/disk-boot_drive-nixos    │ ext4 │ /                   │  24.1 GiB │  21.5 GiB │ false     │ Unknown │
│  1 │ /dev/disk/by-partlabel/disk-nix_store-nix_store │ ext4 │ /nix/store          │ 220.3 GiB │ 172.8 GiB │ false     │ Unknown │
│  2 │ /dev/disk/by-partlabel/disk-nix_store-nix_store │ ext4 │ /nix/store          │ 220.3 GiB │ 172.8 GiB │ false     │ Unknown │
│  3 │ /dev/sdb1                                       │ ext4 │ /mnt/scratch        │ 228.2 GiB │ 215.6 GiB │ false     │ SSD     │
│  4 │ /dev/sdf1                                       │ vfat │ /boot               │   4.0 GiB │   3.9 GiB │ true      │ HDD     │
│  5 │ tank/home-assistant                             │ zfs  │ /mnt/home-assistant │   2.8 TiB │   2.8 TiB │ false     │ Unknown │
│  6 │ tank/mysql                                      │ zfs  │ /mnt/mysql          │   2.8 TiB │   2.8 TiB │ false     │ Unknown │
│  7 │ tank/samba                                      │ zfs  │ /mnt/samba          │   3.3 TiB │   2.8 TiB │ false     │ Unknown │
│  8 │ tank                                            │ zfs  │ /tank               │   2.8 TiB │   2.8 TiB │ false     │ Unknown │
│  9 │ tank/transmission                               │ zfs  │ /mnt/transmission   │   2.8 TiB │   2.8 TiB │ false     │ Unknown │
│ 10 │ tank/nextcloud                                  │ zfs  │ /mnt/nextcloud      │   3.0 TiB │   2.8 TiB │ false     │ Unknown │
│ 11 │ tank/forgejo                                    │ zfs  │ /mnt/forgejo        │   2.8 TiB │   2.8 TiB │ false     │ Unknown │
│ 12 │ tank/ftp                                        │ zfs  │ /mnt/ftp            │   2.8 TiB │   2.8 TiB │ false     │ Unknown │
╰────┴─────────────────────────────────────────────────┴──────┴─────────────────────┴───────────┴───────────┴───────────┴─────────╯

So the savings here might not be as big as you assume.

We already depend on pythonMinimal for some filesystem utilities. Not just the networking script, that I added in nixos-images btw. Adding nushell would increase our image by 50MB (currently at 300MB), which seems like a lot.

Which kind of disk image are you talking about here, and how are you determining the 300MB?

How would nushell compare to pythonMinimal in terms of file size?

Just for fun, I thought I try to find out using nushell itself:

~> nix path-info -r --json 'nixpkgs#nushell' | from json | select path narSize | update narSize { into filesize } | append { path: Sum, narSize: ($in.NarSize | math sum | into filesize)}
╭───┬────────────────────────────────────────────────────────────────┬───────────╮
 # │                              path                              │  narSize  │
├───┼────────────────────────────────────────────────────────────────┼───────────┤
 0  /nix/store/1w90l4fm5lzhlybipfilyjij2das6w98-openssl-3.0.14        6.3 MiB 
 1  /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib        8.7 MiB 
 2  /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52        28.8 MiB 
 3  /nix/store/8hq38sx1hjgganlj6dn0fvvba0g7gcpi-libidn2-2.3.7       352.6 KiB 
 4  /nix/store/b4jwx295fqxg036jxb5bavvd4knikiix-nushell-0.98.0       37.7 MiB 
 5  /nix/store/lw2macnzp3av47m41qv6vcnq2daibgl1-libunistring-1.2      1.8 MiB 
 6  /nix/store/q0iz2x35ki1aaqpjagjfi8s7q053cmc5-gcc-13.3.0-libgcc   155.9 KiB 
 7  /nix/store/x45xiwvk2v97bw1yp1vb6rnmvhagpgyz-xgcc-13.3.0-libgcc  155.9 KiB 
 8  Sum                                                              83.8 MiB 
╰───┴────────────────────────────────────────────────────────────────┴───────────╯
~> nix path-info -r --json 'nixpkgs#python3Minimal' | from json | select path narSize | update narSize { into filesize } | append { path: Sum, narSize: ($in.NarSize | math sum | into filesize)}
╭────┬────────────────────────────────────────────────────────────────────┬───────────╮
  # │                                path                                │  narSize  │
├────┼────────────────────────────────────────────────────────────────────┼───────────┤
  0  /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib            8.7 MiB 
  1  /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52            28.8 MiB 
  2  /nix/store/3hxxbbjc8r66nravvjind6ixhz7cpij1-mpdecimal-4.0.0         217.0 KiB 
  3  /nix/store/4w3c2c11j2kmwxrznji2i7is53z5aldi-xz-5.6.2                807.4 KiB 
  4  /nix/store/8hq38sx1hjgganlj6dn0fvvba0g7gcpi-libidn2-2.3.7           352.6 KiB 
  5  /nix/store/f0dmadrh535bpayz2r4q5lslg0m17lnw-expat-2.6.2             271.0 KiB 
  6  /nix/store/izpf49b74i15pcr9708s3xdwyqs4jxwl-bash-5.2p32               1.6 MiB 
  7  /nix/store/lhfgwjwwdacygiq702yvbahvpqmxivpx-libxcrypt-4.4.36        128.0 KiB 
  8  /nix/store/lw2macnzp3av47m41qv6vcnq2daibgl1-libunistring-1.2          1.8 MiB 
  9  /nix/store/p6mm7p3i4325i2df4l993plgbmscdhlr-libffi-3.4.6             71.9 KiB 
 10  /nix/store/q0iz2x35ki1aaqpjagjfi8s7q053cmc5-gcc-13.3.0-libgcc       155.9 KiB 
 11  /nix/store/qhf8n3srrg0vdd44k7q71fb0p5az15b5-bzip2-1.0.8              78.9 KiB 
 12  /nix/store/rqs1zrcncqz3966khjndg1183cpdnqxs-zlib-1.3.1              125.2 KiB 
 13  /nix/store/saa0d82cwvsy8y8qx3z8lp64gh98a689-python3-minimal-3.12.5   26.3 MiB 
 14  /nix/store/x45xiwvk2v97bw1yp1vb6rnmvhagpgyz-xgcc-13.3.0-libgcc      155.9 KiB 
 15  Sum                                                                  69.4 MiB 
╰────┴────────────────────────────────────────────────────────────────────┴───────────╯

So yeah, the difference is pretty small, but as @Mic92 said we already depend on python for some utilities, and I'm unsure if all of them could/should be rewritten as well.

One nice property of bash besides the really bad error handling is that its execution is very easy to introspect with set -x. So I think whatever replaces the bash code should be also be easy to debug at runtime.

Something like set -x currently doesn't exist for nushell.

It does, however, have very nice built-in logging levels.

If you do decide to give Nushell a chance I'd be happy to assist with a PoC/code migration. While Nix itself is something I'm still learning Nushell has been my daily driver for the past three years and is the main tool used for any sort of automation at my day job

Then maybe you can answer a question: What is the error handling story of nushell? What happens when a command fails? I found create your own errors, which is cool, but I'd like to know how to handle a command that can potentially fail (like invocations of sgdisk).


Note for implementation: parted has a --json flag that may allow for easier parsing of its output than the currently used gptfdisk.

@Yethal
Copy link

Yethal commented Oct 8, 2024

Then maybe you can answer a question: What is the error handling story of nushell? What happens when a command fails? I found create your own errors, which is cool, but I'd like to know how to handle a command that can potentially fail (like invocations of sgdisk).

There are two approaches here. You can wrap the command in try catch block and then if the command fails the catch block will receive the error as closure parameter, or pipe it into complete command which will give you a record with exit_code, stdout and stderr fields which you can then use in either if-else or in a match statement

Also, the store-info size can be shortened to: nix path-info -r --json 'nixpkgs#nushell'|from json|values|get narSize|into filesize|math sum

into commands don't need to be wrapped in update, if the pipeline input is a list you pass it directly, if it's a table you pass the column names you want to convert as rest params

@iFreilicht
Copy link
Contributor Author

Ahh nice, complete is what I was looking for!

Also, the store-info size can be shortened to: nix path-info -r --json 'nixpkgs#nushell'|from json|values|get narSize|into filesize|math sum

Not true, you need to remove values. But yes, that's what I had initially, but I wanted to show a more complete overview of what the closures actually contained.

@Yethal
Copy link

Yethal commented Oct 8, 2024

It seems we're getting slightly different output of nix path-info. I receive a record with nix store paths as keys and the data as values (hence use of values) whereas you get a table (list of records)

As for lsblk if all that's missing are uuids they can be retrieved like this:

  #!/usr/bin/env nu
  ls /dev/disk/by-uuid
  | insert device {|x| $x.name|path expand}
  | update name {path basename}
  | select name device
  | rename uuid
  | join (sys disks) device

Other missing fields can similarly be retrieved by joining appropriate /dev/disk/by- directories to sys disks like this:

#!/usr/bin/env nu
let fields = [id path uuid partuuid]

$fields
| par-each {|field|
    ls $"/dev/disk/by-($field)"
    | insert device {|x| $x.name|path expand}
    | update name {path basename}
    | select name device
    | rename $field
}
| reduce --fold (sys disks) {|it acc|
    $acc
    | join $it device
}

@iFreilicht
Copy link
Contributor Author

That seems to be the case. Could be because I'm running Lix instead of NixCpp.

Indeed, though I feel like that's not a good use of developer time. Using lsblk --json seems much more productive.

@Yethal
Copy link

Yethal commented Oct 8, 2024

Maybe not but it's one less utility to depend on

@phaer
Copy link
Member

phaer commented Oct 8, 2024

I feel like this is at risk of becoming a bit of an academic debate until someone goes and writes a demo with a port of at least some Disko modules to either python, nushell or whatever seems fashionable and does not inflate disk size requirements too much :)

Should be doable and advice is available if someone decides to invest the Time. Only then can we really decide on its merits.

An idea I would like to tryout sometime is to just serialize the resulting Disko options to JSON instead of bash script snippets and parse that JSON in whatever language, be it jq, python or nushell. With a schema sprinkled on top this could help us becoming more flexible in both directions

@Yethal
Copy link

Yethal commented Oct 8, 2024

@phaer I started hacking on it

@iFreilicht
Copy link
Contributor Author

After a lot of experimentation, I've now decided definitively that Python is the way to go.

If you're interested in checking the progress, you can see my (now abandoned) attempt at rewriting in nushell at disko-rewrite-nushell and the current attempt at rewriting in python at disko-rewrite-python (which has feature parity with the nushell version at the time of writing).

First off, I want to note what I liked about nushell for this task

  1. Shell-like. As disko is a program that mainly interacts with shell commands, this is a huge win. It makes the interaction between the code of disko and external shell programs very seamless. Python is quite clunky in this regard, requiring me to write more boilerplate.
  2. Different from bash in all the right ways. The built-in syntax for objects, primitives like complete, well-defined data types and interaction with JSON output makes integrating with even complex tools very easy.
  3. Oh the pipes. Piping in nushell is implemented beautifully. Almost everything is an expression and can be piped to and from, allowing me to write very terse yet expressive code where data flows in the reading direction. In Python I either write nested function calls, which reverses the order of execution, use temporary variables, which is verbose, or use additional libraries for pipe-like flows, which is limited and more confusing than useful in most cases.
  4. Simple and flexible module system. This is great in the context of disko because it allowed me to just place the nushell code alongside the existing nix code without having to move anything. If you compare this to the python code, I had to move large parts of the code base just so all modules could be imported when running from the repo as well as from the package.
  5. Simple type system. In nushell, every object is just that, an object. Similarly simple as JS, but with an actual type checker so you can build consistent interfaces and datastructures with very little effort. Python has a much bigger focus on classes and objects, and while using dicts instead feels very similar on the surface the syntax is much clunkier and you can't type it as well as nushell's record, which allows specifying exact types for each key.

However, there is still a lot of stuff missing from nushell for it to be a sensible choice for this usecase:

  1. More powerful way to write and document commands. Nushell uses comments next to commands and arguments, and this is fine to write simple, well-documented scripts, but disko has multiple modes of operation with overlapping but not equivalent arguments that all need to be checked at runtime for completion. It is currently impossible to have a single source of truth for which values are allowed for an argument and generate documentation, completion and validation from that. Python requires quite a bit of code for this, and while you can't have a single source of truth either, you can at least type-check them against each other.
  2. Proper and complete typesystem. Nushell has no type aliases, no unions, no algebraic types in general. There is a little bit of freedom in pipeline signatures to define multiple variants, but it's not enough to get good compile-time type checking, and gets VERY verbose quickly.
  3. Better developer tooling. I've been using the official extension for vscode, which is probably the best experience you can get, but it's still early days. Highlighting is inconsistent, many LSP functions like "go to definition" and "rename" are not implemented, and this makes writing and analyzing code manual and tedious. Often the LSP also got "stuck" with an error related to another module and I had to reload the entire window to fix it. Another user reported that nushell is currently broken in treesitter, which is not their fault of course, but poses an additional hurdle for potential contributors.
  4. Working autoformatter. nufmt exists, but it has some very strict and annoying rules that completely preclude its use, like not allowing any empty lines at all. It is fast, which is good, but its output is less readable than the code I write by hand, so I consider it unusable. A project with multiple collaborators absolutely needs a good autoformatter. Nushell has none, Python has multiple, many of which have great IDE integration and are very configurable.
  5. Better docs. There's still many pages in the language reference that just contain "TODO". I need better resources for learning and reference. Python on the other hand has incredibly mature documentation written by an army of people, and just websearching for how to solve a problem usually turns up a useful answer.
  6. Better support for the ecosystem in nixpkgs. Right now there are two packages related to nushell in nixpkgs: nushell itself and nufmt. nupm is still in its early days as well, and even if it was mature, nixpkgs would need a whole own mechanism (like python's .withPackages) for package management. I need to write everything myself, or copy/paste code from libraries or the user discord into disko's codebase.

Conclusion

I will be using Python going forward. I'm not opposed to anyone advocating for other languages still, but before you do, please read what I wrote above and think about whether your suggestion would solve the issues I outlined.

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-is-nixpkgs-preferred-programming-language/53848/40

@Sk7Str1p3
Copy link
Contributor

As I see disko is being rewritten in Python?

@iFreilicht
Copy link
Contributor Author

Partially. We just want to replace most of if not all the bash with python, not the nix code. You can check out cli.py on the disko-rewrite-python branch and everything it imports to see the current status.

iFreilicht added a commit that referenced this issue Nov 12, 2024
This will be very useful for generating documentation.

Backported from a5c646b for #789
iFreilicht added a commit that referenced this issue Nov 13, 2024
This will be very useful for generating documentation.

Backported from a5c646b for #789
iFreilicht added a commit that referenced this issue Nov 13, 2024
This will be very useful for generating documentation.

Backported from a5c646b for #789
@Sk7Str1p3
Copy link
Contributor

I read the thread and understood there's a problem with dependencies. I think would be a good idea rewrite Disko in C/C++ because glibc ships with NixOS by default, lang is easy to read and expand and blazing fast.

@iFreilicht
Copy link
Contributor Author

iFreilicht commented Nov 14, 2024

@Sk7Str1pe Sure, feel free to give it a go. You can use my branch(es) for reference. I think you'll find quickly that neither C nor C++ is a good language for this task.

I will be using Python.

mergify bot pushed a commit that referenced this issue Nov 16, 2024
This will be very useful for generating documentation.

Backported from a5c646b for #789
@Mic92
Copy link
Member

Mic92 commented Nov 20, 2024

I read the thread and understood there's a problem with dependencies. I think would be a good idea rewrite Disko in C/C++ because glibc ships with NixOS by default, lang is easy to read and expand and blazing fast.

It's not the glibc that is the problem, but we have to potentially build disko inside a in-memory installer, where we would need to download gcc + glibc.dev + binutils as well as potential libraries. This can make installer go out-of-memory as they need to store the filesystem in RAM.

@Sk7Str1p3
Copy link
Contributor

I checked NixOS ISO, all of those packages are pre-installed, downloading is not the problem. But I see another one - compilation requires quite powerful hardware and especially a lot of memory => we should not compile in the installer and download binaries.

@Mic92
Copy link
Member

Mic92 commented Nov 21, 2024

@Sk7Str1p3 Our kexec image doesn't have any build toolchain installed: https://github.com/nix-community/nixos-images/releases/tag/nixos-unstable
And even if the nixos install iso has a compiler, there is a good chance it wouldn't be used a user uses flakes and nixpkgs channel of that is different from the one that was used to build the iso.

@Sk7Str1p3
Copy link
Contributor

@Mic92 Why we must select toolchain tho?can't we simply use binary cache? This would make usage much faster and less RAM demanding

@Mic92
Copy link
Member

Mic92 commented Nov 21, 2024

This would require Disko to be in nixpkgs so that source code and the module system are kept in sync. Or some sort of versioning.

@Sk7Str1p3
Copy link
Contributor

disko IS in nixpkgs already. how cant you know this if you are a package maintainer???
image

@Mic92
Copy link
Member

Mic92 commented Nov 21, 2024

The disko cli is just a thin layer around the module system. It doesn't really matter what language we write it in, we can just keep it as it is, because shell builds fast and we don't need many features. It doesn't really need to be kept in sync with the underlying module system. What were talking about in this thread is having a different language for the language that consumes the disko module system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

9 participants