Two ways to run Kubernetes on apple/container — Talos (embedded etcd) and k3s (external-etcd HA) #1865
Replies: 3 comments 7 replies
Surely having one VM per node is the standard deployment? Like how minikube did it, before kind. It was Kata containers and now Apple container that went back to the one VM per container model... I thought it would be an interesting concept of having one VM per pod, as a compromise between the two. |
|
You're right, "unusual" was the wrong word — one VM per node is the classic minikube driver model (hyperkit/qemu), and kind was the departure from it, not this. I should have said the substrate is unusual, not the topology. What's actually going on: apple/container's native grain is one micro-VM per container. k3ac aggregates that up to the node — each k3s node is a single micro-VM running containerd, and the pods on it share that node's kernel. So the isolation boundary today is the node, exactly like classic minikube; "no shared kernel" holds between nodes, not between pods on a node. The novelty is just the VM provider (vmnet-native on Apple Silicon, no Docker socket), not the cluster shape. The one-VM-per-pod compromise you're describing is the genuinely interesting direction — and it's essentially Kata's pod-sandbox model. On apple/container that wouldn't be a node provisioner like k3ac at all; it'd be a CRI runtime shim mapping each pod to its own micro-VM — apple/container playing the role Kata's runtime plays under containerd. That actually fits the substrate's grain better than what I'm doing: I'm flattening per-container VMs up into per-node, where that approach keeps the VM boundary at the pod. Bigger project, but you've pointed at the one that fits the runtime best — probably where the first-party #1673 question gets decided. |
|
The VM provider isn't all that novel either, it is the same VZ, the main difference is the image - from cloud image to container image. That also lends itself to pre-installing the software from the Dockerfile, instead of bootstrapping after the first boot. I still don't think there will be enough memory to allocate gigabytes for each pod (or app), and there is the whole CRI runtime support missing as well. So until that materializes, one VM per node is the pragmatic approach. Possibly shared, like kind. |
|
Coming back to this on The first-party path that landed is the I have not re-run k3ac against 1.3.1. The 1.3.1 notes are security / containerization 0.42.0 — nothing that obviously retires the landmines you named. From here they still look like first-party runtime work, not plugin work:
Happy to take one of those as a runtime PR rather than putting k3s inside |
|
Thanks for coming back to this on 1.3.1 — and for offering a runtime PR rather than stuffing k3s into I had not re-run k3ac against 1.3.1 either. I have now, on Homebrew 1.3.1 / Apple silicon, isolated app-root, including a macOS reboot. Stock 1.3.1 does not retire the landmines. One new one showed up. (2) is still the worst, and the one I would rather see first: a reserved lease / Stop/start still rotates vmnet DHCP. On this machine a k3s server went Nearby (1) (3) I would not take as a runtime PR. After this reboot (boot 2026-09-06 21:37) I did not re-run One 1.3.1 regression k3ac did hit, not in your list: If you pick one runtime change: |
|
Thanks for re-running k3ac on 1.3.1 and ranking these. I'll take (2). Retracting (3): you are right — on 1.3.1 the domain survives reboot and #2145 is still open against #2120 and is not in 1.3.1. The I can file a dedicated issue for the |
|
A few data points from kiac (multi-node kubeadm and k3s on apple/container since June), limited to what this thread has not covered: IP rotation. +1 to sticky addressing (#1740) as the runtime fix. Until then kiac recovers on resume by rewriting every address-bearing file (admin/controller-manager/scheduler/kubelet kubeconfigs and the static pod manifests), re-issuing the apiserver serving cert with MetalLB on vmnet. Correction (see my reply below): re-tested on 1.3.1, host TCP to a floating VIP works; the drop described here did not reproduce and should be disregarded. One caveat on "gratuitous ARP works", from kubeadm clusters on container 1.0.0 in June: a floating VIP answered ARP but the host's TCP to it was dropped (node-to-node VIP traffic worked), so kiac used node IPs as the pool. That exposed a second problem: with one shared L2 pool and a cluster-wide L2Advertisement, a speaker on a non-owner node could win the ARP election for a live node IP and hijack that node's traffic, apiserver heartbeats included, taking it NotReady minutes after the first LoadBalancer Service (saiyam1814/kiac@2a65ec7 pins advertisements per node). kiac later replaced MetalLB with a small shell controller that assigns node IPs to Services, no speaker and no images. Different distro and an older release than your Talos run, so it may not apply there. One more landmine (container 1.0.0, kernel 6.12.28; not yet re-run on 1.3.1). A 1 MiB TCP upload from one apple/container VM to another stalls after 170-330 KB when the receiving VM forwards the stream across a veth into another network namespace (a NodePort, or any published port into a pod), only while the sender has TSO on. NAT is not the trigger: DNAT with local delivery is fine, plain forwarding into the veth stalls, and the macOS host sending through the identical DNAT+veth path moves 1 MiB in 7 ms. Receiver-side offload settings and an MSS clamp change nothing; the sender's TSO super-frames arrive as single 2.5-6 KB IP packets on a 1280-MTU interface and forwarding them into the 1500-byte veth fails with IpFragFails, which is what an skb without GSO metadata does (inferred from the counters, details in the issue). Sender
|
|
Thanks — that is useful, especially the TSO forwarding stall as its own runtime bug. Sticky addressing stays first. I will start from #1740 rather than another overlapping PR. The kubeconfig/cert rewrite is a solid provisioner fallback until a reserved lease exists; I am not proposing to put that recovery into #2245 is the right place for the sender-TSO / veth-forward stall. I will not mix it with #1873 or #1927, and I will not take it as the first runtime change. Same |
|
Following up on the MetalLB point, since you left it as a data point pending 1.3.1: I re-tested it today and I withdraw the TCP part. Setup: container CLI 1.3.1, macOS 26.2, default kernel 6.12.28, default network. Two-node kubeadm 1.37 cluster (kindest/node, kindnet), MetalLB v0.16.0 native manifest, one
So on 1.3.1 the floating VIP resolves via ARP and host TCP to it works, which matches your Talos result; the only oddity is that ICMP echo to the VIP gets no reply while TCP does. What kiac saw in June on 1.0.0 was either version-specific or a misread of the second problem, which was a MetalLB configuration mistake on our side (a shared pool of node IPs with a cluster-wide L2Advertisement let a speaker announce another node's live address; that is what saiyam1814/kiac@2a65ec7 fixed, and it was never a vmnet property). Today's single-pool test does not exercise that scenario. Please treat my earlier "host TCP to a floating VIP was dropped" as not applicable to 1.3.1. |
|
Thanks for re-running that on 1.3.1. I will treat the June host-TCP-to-VIP drop as not applicable here. ICMP silence on the VIP with working TCP is not a reason to change the sticky-IP plan. |
Uh oh!
There was an error while loading. Please reload this page.
Apple's
containerruns each container as its own micro-VM with its own vmnet IP. That's an unusual base for Kubernetes — one real VM per node, no shared kernel, no Docker socket. I wanted to know what it can actually host, so I built two small provisioners on it and ran them on hardware. Sharing what the substrate allows, and the landmines.k3ac) — external etcd quorum + haproxy HA: https://github.com/BinHsu/aegis-apple-container-provisioner-k3sBoth boot one micro-VM per node and form a real multi-node cluster with per-node IPs. They are proven spikes, not maintained tooling — the code is the receipts (each repo carries an ADR trail and a first-person hardware verification log). Credit where due: @saiyam1814's
kiacalready explored the kubeadm path; these are the Talos and k3s data points alongside it.Where the two diverge — HA, and it's the interesting part:
--cluster-initforms the cluster after boot, and embedded etcd's peer membership is IP-bound — cold-restart the VMs, vmnet hands out new DHCP IPs, the members look for peers at the old addresses, and quorum never reforms. So k3ac uses k3s's pluggable external datastore: a 3-node etcd quorum (mutual TLS) addressed by FQDN, fronted by an haproxy L4 load balancer. The cluster reconnects by name after any DHCP shift, and it adds day-2 ops (snapshot/restore, rolling upgrade/rollback, cert/token rotation), all behind a-forcedry-run gate.Substrate gotchas (all verified on hardware), shared by both lanes:
container cpfaults the vsock during k3s/Talos cold-boot image extraction → deliver the kubeconfig by host bind-mount instead.The Talos one I pitched upstream; it was declined on principled grounds (siderolabs/talos#13587), so it stays as a dev/CI substrate.
Write-ups — the DHCP traces, the ADRs, the maintainer's reasoning:
Curious whether a first-party Kubernetes story (discussion #1673) lands on k3s or something else.
All reactions