Overlaying Rancher k3os persistent directories on a unified overlayfs dedicated auxiliary block device

[j-landru]

Is your feature request related to a problem? Please describe.
Context
For lab usage I plan to deploy a k8s cluster based on Rancher K3os/k3s composed of several nodes types (reusing old servers of different generations as well as qemu/kvm VMs). Whatever the number of cluster nodes, I want them booting from a single unique rootfs reference image (baremetal nodes will boot from USB Key or PXE, KVM VMs from a cloned qcow2 derived from that reference image at bootup. Each cloned node will share the same unique rootfs image. I don't want to individually update or upgrade OS on each clone, even using some great operator tools like Ansible, Salt, Chef or Puppet... When an OS update or upgrade is needed, I just want to create and test a new reference rootfs image from a dedicated VM and then stop and restart each clone from that new image. As this means that the rootfs of each node is ephemeral, I want to overlay persistent node data (node parameters, config files, logs, data, ...) on an auxiliary "delta" attached block device unifed to the root filesystem (using aufs or overlayfs) at bootup. Each cluster node works in the same way as a testing or demo Linux distribution booted from a RO LiveDVD or USB key and keeping personalization and data files on a auxiliary USB key.
Adding a new worker node to the cluster just needs to generate a new "delta" preconfigured block device or image and attach it to the node before booting it up. In the future, this could, probably, be done in a CI/CD mode, to dynamically size up the cluster.

Describe alternatives you've considered
According to Rancher k3os doc k3os rootfs is an ext4 block device labeled "K3OS_STATE", the main idea is to label the "delta" auxiliary block device "K3OS_DELTA" and test its presence at bootup to mount it on /k3os/delta dedicated mountpoint on the rootfs and then dynamically overlayfs (or aufs) mount the persistent k3os directories. The best way would probably to detect "K3OS_DELTA" device and mount a global unified filesystem root at the end of the initrd sequence, just before chrooting to the new root and launching init process.

At this stage, as I am not able to personalize the k3os initrd, an alternative iniial "PoC" way is to script that early filesystem unification at k3os init_cmd or boot_cmd phase using write_files, boot_cmd, k3os:modules k3os facilities embedded inside the config.yaml file or config.d directory on the master rootfs image. Others k3os cluster detailed configuration could be kept inside the second level config.yaml file or config.d directory embedded on the delta image.

I published some more detailed personal notes as an howto on an alternate gitlab portal : https://framagit.org/-/snippets/5742

Drawbacks of that initial PoC

• needs two master rootfs reference images, one for k3s server (control plan) nodes and another for the k3s agent (worker) nodes.
• an ugly recursive boot_cmd script call to relay boot sequence from the master disk to the delta disk. This recursive call introduce a bootloop risk which is painful to debug.
• containerd issue quick an dirty hacked : pods starting issue, k3s/containerd use ovelrlayfs to snapshot images when starting pods, and doesn't seem to like a two level ovelrayfs fs (nested overlayfs), as stated in rancher/k3s Switch niusmallnan to rancher #19 issue at github (see Failed to startup pod after "re-install" k3s-io/k3s#19 (comment)). *Quick and dirty hack : move /var/lib/rancher/k3s/agent dir elsewhere (/k3os/delta/var-lib-rancher-k3s-agent dir) outside the overlayed dirs (home, var, opt), and soft link /k3os/delta/var/lib/rancher/k3s/agent pointing to that new place.

Describe the solution you'd like
As stated above, the neater way would probably to detect "K3OS_DELTA" block device at the end of the initrd sequence, and unify the whole filesystem just before pivotchrooting to the new root and launching init process. That way no need to make ugly recursive call of boot scripts.
And the ultimate goal would be to run server nodes and agent nodes directly from the k3os iso image with no initial installation on a server or agent master reference image. Probably need to find a way to initiate the k3s cluster on the 1st k3s server "K3OS_DELTA" image with an appropriate k3s server call.

Nota still need to find a workaround for the containerd nested overlayfs issue