Thoughts on hobby clusters... + ansible-playbook

[alexpdp7]

I think Talos is great for personal infra. I've been looking at Fedora CoreOS, but Talos looks like a simpler way to get a cluster running in a "full-stack" fashion; management of the OS and the cluster in one go.

I'm thinking about a single node running control plane and worker, and perhaps some extra worker (my use case for this is having an ARM worker for CI/CD).

However, I think the following pieces would go a long way towards supporting hobby users (from high to low):

• feat: dhcpv4: send current hostname, fix spec compliance of renewals #5897 : this would make things a little easier, and not require me to manage MAC addresses or deal with cloud-init.
• Document how you are supposed to store the cluster config. I think it's mostly suitable to store in Git, like the rest of your configuration-as-code, but I was a bit bothered that even with talosctl gen secrets, there seemed to be private keys in the rest of manifests.
• Document how to configure "lightweight" local storage. Ceph/Rook might be a bit too demanding for hobby clusters. Hostpath, TopoLVM, etc. may be sufficient for hobby clusters. (Due to the sad demise of Promscale, my cluster does not need storage for now, so I wouldn't be working on this.)
• Move to the official docs how to make the control plane schedulable ( Can one machine be both control-plane and worker node? #5101 ). docs: add how tos for workloads on control planes, and scaling up #6885
• Personally, document some opinionated "known-good" fuller stack (I like kube-vip as a load-balancer, because it's DHCP-aware; I also like haproxy-ingress, and exposing core-dns as a node port...)

So, except for the DHCP hostname issues, IMHO having a great "hobby" experience with Talos is only a matter of... documentation? I'm probably going to publish some documentation on that as I try to set up my cluster- but if there's interest, I wouldn't mind trying to do that as PRs to the docs (but really, before putting the effort, I'd like to know if there's a good path to getting those merged).

(Given the Proxmox instructions, I feel that Talos has in its priorities a good "hobby" experience. I do think that people that have successful experiences with Talos for personal purposes might recommend purchases at their jobs...)

[alexpdp7]

Well, I got a first version of my Ansible role that deals with some of that stuff:

https://github.com/alexpdp7/alexpdp7/tree/master/personal_infra/playbooks/roles/talos

It's very crummy and hacky (only does single-node clusters for now, only on Proxmox, doesn't deal great with failures...). I'm keeping the secrets file in the repo, vaulted, everything else is regenerated on the fly. Due to not setting a DHCP client id, I need to do some extra stuff to get things working fully automatically (without having to manage MACs, etc. manually). I let the host boot, then peek the IP from the DHCP server.

[steverfrancis]

We'd love PRs for the docs, thanks!
I did create a PR to address the making a control plane schedulable, which is awaiting merging: #6885

But we'd love others!

[alexpdp7]

Ah, great news!

So is my current approach for keeping the cluster config in a repo valid?

• Manual talosctl gen secrets
• Encrypt that file (I use ansible-vault) and store it in the repo
• Generate patch files using my configuration management (because I want to inject variables from my infra-as-code configuration) on-the-fly, not storing them in a repo (because they are generated files)
• When I need the other Talos configuration files, I also generate them on the fly, not storing them in the repo (because they are generated files and they contain secrets).

I think it's valid, but maybe it's not "optimal"? It seems that the host configuration is deterministic, but talosconfig is not deterministic.

If this is something that would be accepted into the docs, then I'm happy to flesh it out and format it a little.

[alexpdp7]

OK, I have polished my Ansible role and documented it a bit:

https://github.com/alexpdp7/alexpdp7/blob/master/personal_infra/playbooks/roles/talos/README.md

I think it's nifty, because in less than four minutes, the role creates the VM in Proxmox and sets up a cluster. It's still a single-node cluster, but it works fairly well. (It can even manage multiple clusters.)

I think the docs should be something like this:

Configuration management of Talos clusters

Talos lends itself well to deploying clusters using infrastructure as code.

The following steps describe a procedure to maintain clusters using infrastructure as code.

1. Use the talosctl gen secrets command to generate the secrets for a cluster.
   This file contains sensitive information.
   If you keep this file in a source repository, then encrypt the file.
2. Use the talosctl gen config -t controlplane command to derive a control plane configuration file from the secrets file.
   Use a patch file to inject networking configuration, node configuration, etc.
   Keep the patch file in version control, but not the derived file (it is a generated file and it contains sensitive information).
3. If you require worker nodes, repeat the preceding step, adapted to worker nodes.
4. Apply the configuration to the nodes.
5. Use the talosctl gen config -t talosconfig command to derive the talosconfig file.
   Do not store the talosconfig file in version control (it is a generated file and it contains sensitive information).
   Use the talosctl config endpoint command to inject the endpoint in the configuration file.
6. Use the talosctl bootstrap command to bootstrap the cluster.
7. Use the talosctl kubeconfig command to create a kubeconfig file to access the cluster.

The secrets file and configuration patches describe a cluster deployment completely.

For an example of this strategy, refer to this sample Ansible playbook for creating Talos clusters in Proxmox.