Kubernetes control-plane restart fails after node IP rotation

[mxschmitt]

Summary

A single-node Kubernetes cluster created by the Apple container K8s plugin
does not restart after its VM receives a different IPv4 address. Address
rotation is expected; the Kubernetes provider does not preserve the kind
bootstrap contract or configure an address-independent in-VM API endpoint.

Environment

• macOS on Apple Silicon
• apple/container 1.3.0 (d6de569)
• kindest/node:v1.35.5
• One VM with both control-plane and worker roles
• No workload or custom DNS configuration

Minimal reproduction

container k8s create --name k8s-repro
container exec k8s-repro cat /kind/old-ipv4
container stop k8s-repro
container k8s start --name k8s-repro

On a clean runtime, the first ordinary restart changed
192.168.64.2 -> 192.168.64.3 and failed before the K8s plugin could perform
its readiness checks:

INFO: detected IPv4 address: 192.168.64.3
INFO: detected old IPv4 address: 192.168.64.2
error: unable to read config from "/kind/kubeadm.conf":
open /kind/kubeadm.conf: no such file or directory

Root cause

There are two independent restart requirements:

1. The plugin runs kubeadm with /etc/kubernetes/kubeadm-config.yaml, while a
   kind node expects its provider to create /kind/kubeadm.conf for restart
   reconciliation. The kind-owned file was never created; kubeadm did not
   consume or delete it.
2. Without controlPlaneEndpoint, kubeadm persists the VM address in clients
   such as admin.conf and kube-proxy. In an ablation test, adding only the kind
   file let kind repair API-server state, but kube-proxy continued dialing the
   retired address and CoreDNS remained unready.

Expected behavior

The node should recover after intentional address rotation without requiring a
sticky IP. kind should reconcile address-dependent advertise/certificate
state, while persistent clients use an address-independent endpoint.

Validated fix

PR #2158 writes and uses /kind/kubeadm.conf directly and configures
controlPlaneEndpoint: 127.0.0.1:6443 for clients inside the current one-VM
cluster. After rotation, kind rewrites the node advertise address to the new
VM address while in-VM clients retain loopback. Host kubeconfig export remains
unchanged.

The fix is intentionally limited to newly created single-node clusters. It
does not add sticky addressing, DNS, a proxy, broad start-time rewriting, or
migration for clusters created by older releases. A future multi-node topology
will need a stable shared DNS, VIP, or load-balancer endpoint.

[mxschmitt]

Superseded prototype

The earlier fix/k8s-restart-ip-change prototype and its broad start-time
address rewriting have been superseded by PR #2158. Clean-room and ablation
testing showed that the rewriting could not migrate a stock cluster because
kind fails before the plugin regains control; it was also unnecessary for new
fixed clusters.

The final patch is one focused, signed commit. It writes and uses kind's
/kind/kubeadm.conf directly and gives the current single-node cluster a
stable in-VM control-plane endpoint. The prototype commits referenced by the
original version of this comment should not be used for review.

[mxschmitt]

Final clean-room and ablation result

A clean stock 1.3.0 cluster with no workload, address bumper, or custom DNS
failed on its first ordinary restart (192.168.64.2 -> 192.168.64.3) because
kind could not read /kind/kubeadm.conf.

Adding only that file passed kind's first gate, but admin.conf and kube-proxy
still used the retired address. kube-proxy logged no route to host, CoreDNS
remained unready, and k8s start could not complete. Both the kind bootstrap
contract and an address-independent client endpoint are therefore required.

PR #2158 now implements only those two requirements:

• write and use /kind/kubeadm.conf directly;
• use 127.0.0.1:6443 for clients inside the current one-VM cluster.

The rebased, GitHub-Verified commit 86735d0 passed the focused unit tests and
the exact integration regression twice. The final run rotated
192.168.64.5 -> 192.168.64.7, verified the loopback endpoint, verified that
kind rewrote advertiseAddress to the new VM address, completed k8s start,
and observed the node Ready.

The regression uses an intervening Alpine allocation to advance the rotating
allocator; it does not consume the released node address. Existing clusters
created without the kind file are not claimed to be migrated.

[guanchzhou]

Independent reproduction on container 1.3.1 (release, Apple Silicon). Same kube-proxy half of this bug is still present on a cluster that was created days earlier and then stopped/started after the node IP rotated (192.168.64.5 → 192.168.64.3).

What we observed that 1.3.1 does rewrite

After container start k8s-dev + container k8s start --name k8s-dev, these on-disk files already pointed at the new address:

• /etc/kubernetes/kubelet.conf
• /etc/kubernetes/controller-manager.conf
• /etc/kubernetes/scheduler.conf

kubectl from the host also worked once the serving cert was restored (see below). The control plane looked healthy from outside.

What it misses: the in-cluster kube-proxy ConfigMap

kubectl get cm -n kube-system kube-proxy -o jsonpath='{.data.kubeconfig\.conf}'
# still: server: https://192.168.64.5:6443

kube-proxy then loops on:

dial tcp 192.168.64.5:6443: connect: no route to host

It never syncs, so there are no Service rules at all. Every pod call to https://10.96.0.1:443 times out. That is easy to misread as an application bug.

In our case Envoy Gateway certgen logged generated certificates and then:

failed to get server groups: Get "https://10.96.0.1:443/api": dial tcp 10.96.0.1:443: i/o timeout

A busybox pod against https://10.96.0.1:443/version timed out the same way. After patching the ConfigMap to https://192.168.64.3:6443 and kubectl rollout restart daemonset/kube-proxy -n kube-system, the same pod retrieved v1.37.0.

Related: serving cert deleted, not regenerated

On the same restart, kube-apiserver crash-looped on:

open /etc/kubernetes/pki/apiserver.crt: no such file or directory

CA and the rest of /etc/kubernetes/pki survived. The stored kubeadm config still had advertiseAddress: 192.168.64.5 while the live node was .3. The plugin rewrote the apiserver manifest advertise-address to .3 and dropped the stale serving cert, but did not regenerate it. Workaround inside the node:

sed -i 's/192.168.64.5/192.168.64.3/' /etc/kubernetes/kubeadm-config.yaml
kubeadm init phase certs apiserver --config /etc/kubernetes/kubeadm-config.yaml

Environment

• container CLI version 1.3.1 (build: release)
• existing single-node cluster k8s-dev (kindest/node v1.37.0, 2 CPU / 4 GiB at the time)
• macOS Apple Silicon
• no custom DNS

PR 2158's 127.0.0.1:6443 in-VM client endpoint would have avoided the kube-proxy ConfigMap trap for new clusters. Existing clusters created without /kind/kubeadm.conf still hit this on restart, matching the "not claimed to be migrated" note on that PR.