Question - How do you provision your personal computer ?

[bgaillard]

What exactly are you trying to do?

Hi, first thank's a lot for this great tool !

This issue is not directly linked to chezmoi but I decided to open it here because i'm sure several users would be able to help (do not hesitate to close this issue if you think it's definitely not the right place to open it).

I'm using chezmoi since several months and i'm happy with it to manage my dot files. This is great tool to quickly get my configuration files when i'm installing a new computer, ok for this part.

But when my dot files are successfully imported the applications I used to have on my personal laptop still have to be installed (and the list of the apps to install can be very long). I'm looking for an easy solution to provision all the apps I use on a new machine very quickly.

Ideally when I setup a new machine I would like to be able to execute only few commands to have all my dot files and applications installed.

1. Install chezmoi and pull my dot files (ok for this part)
2. Get a configuration file (for example a YAML file) which describes what are all the applications I'm used to have. Have a tool to read this configuration file and provision all those applications in one shot

For example we could have a YAML file which looks like this one (it's only an example and the format is obviously not "very well-designed").

# Languages to install
languages:
    python:
        version: 3.9
    go: 
        version: 1.17.3
        
# Packages to install
packages:

   # Package to be installed with Debian 'apt'
    apt:
        - alsa-utils
        ...
        - xorg
        
    # Packages to be installed with Python pip
    python:
        - aws-shell
        
    # Packages to be installed with custom provisioners
    custom:
        - terraform

This YAML file would allows to describe the following stuffs.

• Which languages to install (useful for developers)
• Which standard distro packages to install
• Which "other kind of packages" to install, for example Python, Go, Rust, Docker, Podman, Snap packages

Ideally "the tool", let's assume we name it thetool would be

• simply called with the configuration file : thetool -f config.yaml
• a simple binary (created in Go or Rust for example) with no complex setup (only the binary + the config.yaml file would be required to use it).

Do you know any tool which allows to do that ? If not how do you re-install your computers to not lost all the apps you're used to install ?

What have you tried so far?

For now I started to develop custom bash scripts which can be easily pulled on a new machine. It works well but it's a custom made solution and is not shared with the community.

Where else have you checked for solutions?

I searched on Internet but did not found any simple solution for a personal computer (most of the tools are dedicated to server provisioning).

• We can think about server provisioning solutions but I've the feeling they are too flexible and overkill : Ansible, CFEngine, Salt, Puppet, Terraform, etc.
• Top 6 Open Source Linux Server Provisioning Software

[twpayne]

Personally I use a provisioning script (Linux and macOS), and have similar approaches for Windows elsewhere.

[bgaillard]

Thanks for your response @twpayne and thanks for sharing your scripts 👍

I did not read chezmoi documentation thoroughly enough because I did not know we could chain several scripts as you did.

After much more research I think what i'm looking for does not really exist but there's probably good reasons. What I'm thinking about would probably not be flexible enough.

For example in your scripts you install eksctl with those commands.

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl completion zsh | sudo tee /usr/local/share/zsh/site-functions/_eksctl > /dev/null

A "perfect world" would only require to add this in a yaml file and it would be transparent to the user.

packages:
    # Packages to be installed with custom provisioners
    custom:
        - eksctl

If i've time and it appears it could be useful to the community i'll try to develop a proof of concept.

[twpayne]

A "perfect world" would only require to add this in a yaml file and it would be transparent to the user.

packages:
    # Packages to be installed with custom provisioners
    custom:
        - eksctl

If i've time and it appears it could be useful to the community i'll try to develop a proof of concept.

If I understand this post correctly, then the intention here is to create some kind of universal package manager, where eksctl (in this example) would be installed using an appropriate method depending on the underlying system.

This is an ambitious goal. For such a universal package manager to be included in chezmoi, it would need to work across all operating systems and architectures supported by chezmoi. For example, it would need to cope with different packages having different names in different systems (RedHat- and Debian-based systems often have different package names for the same software), multiple package managers being available in the same system (e.g. apt and snap being available on Ubuntu-based systems, and Chocolatey, Scoop, and WinGet being available on Windows). Furthermore, the meaning of a package will vary from packaging system to packaging system. For example package foo on system X might be split into separate foo-client and foo-server packages on system Y.

The above reasoning is explicitly why chezmoi currently has no built-in support for cross-platform package management, and instead defers package management to the user through the script mechanism, as @nandalopes so neatly demonstrates.

If you want to work on this, I would suggest researching how whole system management tools support cross-platform package installation (e.g. Puppet's package resource type and Ansibles ansible.builtin.package). Note, of course, that chezmoi scripts can invoke third-party tools like Puppet and Ansible.

Given the above, I would strongly recommend using @nandalopes's approach instead.

[bgaillard]

Thanks @twpayne, I agree with what you described, especially on the fact that the responsibility of chezmoi should stay as is and that this "package manager" (or "meta package manager" don't know how to call it) is an other responsibility.

I'm conscient the work would be huge especially if we would like to support several operating systems and kind of package managers. I think what i'm dreaming about should "only" be a program which delegates it's install responsibilities to other packages managers in most cases (otherwise we would have to manage complex stuffs like dependency management, caching, etc.... it would be too ambitious for a first version).

In fact this is exactly your advice with your links to Puppet and Ansible. Before coding this program I'll need to create a software specification to describe what is a "package" and then think about how to create a "proxy" (as described in the Ansible documentation) to delegation installs to other package managers available on the system.

For the operating systems and package managers diversity I think the tool could ignore this diversity, that's too say the first version of the YAML file could be only for one OS at first (perhaps i'm wrong here, this will be to be validated).

I like @nandalopes approach also and thinks it's good for "advanced users" (advanced because it requires "lot of" scripts and bash scripting).

But i'm thinking about something a little more simple for users, that's why I would like something with only one YAML file to describe "a state" to reconcile with the program.

Finally here i'm perhaps dreaming too much because I have a family and children so i'm not sure i'll have time to work on this efficiently. If i'm crazy i'll let you know when a first version of this tool is ready in alpha / beta. 😆

[twpayne]

I've converted this to a discussion as it has no clear resolution.

[bgaillard]

Sure no problem :)

[nandalopes]

I've only Ubuntu machines to manage, and I use this approach:

• Create application lists templates
• Create run_onchange_after scripts for each package type, and include comments with sha1sum of the application lists
  1. Configure extra sources.list to apt
  2. Install / uninstall apts
  3. Install snaps
  4. Install pips
  5. Install flatpaks

I usually help my friends with transition to Linux, and for that I've created scripts with application profiles, like teacher, writer, movies, gamer, etc.
https://github.com/nandalopes/dotfiles/tree/friends

[bgaillard]

Hello @nandalopes, thanks for those advices. Lot of work on your repo it's awesome 👍

Good idea to use run_onchange_after scripts, what you described very is similar to what i'm thinking about. I dream about a "universal package manager" as @twpayne described before, it would be a little more easy to setup as what you did (just on Go binary and only one YAML file to describe what to install).

Anyway thanks for your help and advices, if I have motivation to start a work of this "package manager" it would help me.

[nandalopes]

Hello @nandalopes, thanks for those advices. Lot of work on your repo it's awesome +1

Good idea to use run_onchange_after scripts, what you described very is similar to what i'm thinking about. I dream about a "universal package manager" as @twpayne described before, it would be a little more easy to setup as what you did (just on Go binary and only one YAML file to describe what to install).

Anyway thanks for your help and advices, if I have motivation to start a work of this "package manager" it would help me.

Hi, there's helpers to deal with JSON files, and you can use xargs on
Unix like OSes to call package manager.

I think it's easier to make a template file with one program per line, and
then read it on a basic run_onchange_after script.

[twpayne]

You might want to consider asdf which addresses many of the problems mentioned above.