kubectl logs/exec/proxy returns 502 Bad Gateway for all worker-node pods; control-plane node works; kubelets healthy and certs consistent

[IT-jamf]

Hoping someone can sanity-check a 502 I can't fix on k3s v1.33.3+k3s1 (4 nodes, 1 server + 3 agents, Debian 13, default flannel, all on one LAN).

kubectl logs/exec for any pod on a worker node returns:
proxy error from 127.0.0.1:6443 while dialing 192.168.100.165:10250, code 502: 502 Bad Gateway
but works fine for pods on the control-plane node. Affects a 305-day-old pod, so not workload-specific; started around a server restart.

• kubelet healthy + reachable (curl to :10250/healthz returns 401 from server and other nodes)
• serving cert valid (curl --cacert returns 401, no cert error; SAN has the node IP)
• not networking/firewall (direct curl to :10250 works from the server)
• not a CA mismatch (worker client-kubelet issuer == server client-ca subject, same timestamp)
• not bind-address/Not able to connect to kubelet when using bind-address flag #10444 (kubelet listening on *:10250, no node-ip/bind/kubelet-arg override)

Server log also shows recurring authentication.go: Unable to authenticate the request: x509: certificate signed by unknown authority around the same time — not sure if that's the tunnel auth or unrelated.

👇👇👇👇 Logs below 👇👇👇👇

# Placeholders: <SERVER_IP> = control-plane node, <WORKER1_IP>/<WORKER2_IP> = agent nodes,
# <POD_IP> = a flannel pod IP, <FINGERPRINT> = cluster cert SHA1.
# The routine "Updating TLS secret for kube-system/k3s-serving" line repeats every ~10h
# from Jun 18 onward and is collapsed to a single example below.

# --- First errors, at/just after the Jun 17 server start ---
Jun 17 18:06:30 server1 k3s[858]: level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:51768: dial tcp <POD_IP>:10250: connect: no route to host"
Jun 17 18:06:30 server1 k3s[858]: level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:51800: dial tcp <POD_IP>:10250: connect: no route to host"

# --- Routine cert rotation, repeats ~every 10h Jun 18 -> Jun 29 (collapsed; one example) ---
Jun 18 04:06:22 server1 k3s[858]: level=info msg="Updating TLS secret for kube-system/k3s-serving (count: 10): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-<SERVER_IP>:<SERVER_IP> listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-server1:server1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=<FINGERPRINT>]"
# ... (same line repeats every ~10h through Jun 29; omitted) ...

# --- etcd compaction, routine ---
Jun 26 09:01:12 server1 k3s[858]: level=info msg="COMPACT compactRev=23809068 targetCompactRev=23809250 currentRev=23810250"
Jun 26 09:01:12 server1 k3s[858]: level=info msg="COMPACT deleted 182 rows from 182 revisions in 1.902144ms - compacted to 23809250/23810250"

# --- Authentication errors (Jun 29) ---
Jun 29 09:47:41 server1 k3s[858]: E0629 09:47:41.364683 858 authentication.go:75] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=3634531073, SKID=B0:BB:C0:B3:77:08:6C:1B:04:7B:F1:D0:2D:8F:E9:0E:0F:1C:21:26, AKID=E3:22:D2:25:4D:C6:8F:0E:86:39:40:93:B0:0C:E9:78:8E:F6:46:97 failed: x509: certificate signed by unknown authority]"
Jun 29 09:47:55 server1 k3s[858]: E0629 09:47:55.098495 858 authentication.go:75] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=3634531073, SKID=B0:BB:C0:B3:77:08:6C:1B:04:7B:F1:D0:2D:8F:E9:0E:0F:1C:21:26, AKID=E3:22:D2:25:4D:C6:8F:0E:86:39:40:93:B0:0C:E9:78:8E:F6:46:97 failed: x509: certificate signed by unknown authority]"

# --- 502 proxy errors dialing WORKER kubelets (Jun 29) ---
Jun 29 10:31:19 server1 k3s[858]: E0629 10:31:19.086463 858 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: Get \"https://<WORKER1_IP>:10250/containerLogs/<ns>/<pod>/<container>\": proxy error from 127.0.0.1:6443 while dialing <WORKER1_IP>:10250, code 502: 502 Bad Gateway"
Jun 29 10:32:41 server1 k3s[858]: E0629 10:32:41.667673 858 status.go:71] "Unhandled Error" err="... Get \"https://<WORKER1_IP>:10250/containerLogs/<ns>/<pod>/<container>\": proxy error from 127.0.0.1:6443 while dialing <WORKER1_IP>:10250, code 502: 502 Bad Gateway"
Jun 29 10:32:49 server1 k3s[858]: E0629 10:32:49.657587 858 status.go:71] "Unhandled Error" err="... Get \"https://<WORKER1_IP>:10250/containerLogs/<ns>/<pod>/<container>\": proxy error from 127.0.0.1:6443 while dialing <WORKER1_IP>:10250, code 502: 502 Bad Gateway"
Jun 29 10:34:34 server1 k3s[858]: E0629 10:34:34.552352 858 status.go:71] "Unhandled Error" err="... Get \"https://<WORKER1_IP>:10250/containerLogs/kube-system/svclb-traefik-XXXX/lb-tcp-80?tailLines=5\": proxy error from 127.0.0.1:6443 while dialing <WORKER1_IP>:10250, code 502: 502 Bad Gateway"
Jun 29 10:37:14 server1 k3s[858]: E0629 10:37:14.275479 858 status.go:71] "Unhandled Error" err="... Get \"https://<WORKER1_IP>:10250/containerLogs/kube-system/svclb-traefik-XXXX/lb-tcp-80?tailLines=5\": proxy error from 127.0.0.1:6443 while dialing <WORKER1_IP>:10250, code 502: 502 Bad Gateway"
Jun 29 11:29:09 server1 k3s[858]: E0629 11:29:09.024586 858 status.go:71] "Unhandled Error" err="... Get \"https://<WORKER2_IP>:10250/containerLogs/<ns>/<pod>/<container>\": proxy error from 127.0.0.1:6443 while dialing <WORKER2_IP>:10250, code 502: 502 Bad Gateway"

[IT-jamf]

@brandond I'd be extremely grateful if you can take a look at this.

[brandond]

Start by upgrading to the latest 1.33 patch release. The release you are on is almost a year old.

If the problem continues, then we can do additional research.

[MentalOfCrow]

The direct curl <worker>:10250 test is useful, but it does not exercise the path that K3s uses by default. In the default egress-selector-mode=agent, the apiserver reaches a remote kubelet through that node's reverse WebSocket/remotedialer session. K3s rewrites the remote kubelet destination to loopback inside the selected agent tunnel. That also explains why a local control-plane pod works while all worker pods fail.

So I would treat this first as missing/rejected worker tunnel sessions, not as a kubelet listener/firewall problem. The x509: certificate signed by unknown authority messages appearing after the server restart are highly relevant because the tunnel WebSocket is authenticated with the agent's kubelet client credentials.

1. Confirm the configured path

On the server:

sudo grep -R "egress-selector-mode" \
  /etc/rancher/k3s/config.yaml /etc/systemd/system/k3s.service* 2>/dev/null

sudo cat /var/lib/rancher/k3s/server/etc/egress-selector-config.yaml

If nothing overrides it, agent is the default.

2. Verify that every worker actually has a live tunnel

On each worker, around a fresh agent restart:

sudo journalctl -u k3s-agent -b --no-pager | \
  grep -Ei 'remotedialer|tunnel|websocket|proxy|x509|certificate|6443'

A healthy current agent logs a message equivalent to:

Remotedialer connected to proxy

The K3s source builds this connection as:

wss://<server>/v1-k3s/connect

and reconnects after an authentication or transport failure. If the three workers lack the connected message or show a reconnect loop, that is the direct cause of the 502s.

3. Identify the exact rejected certificate

The server log already gives a Subject Key Identifier. On each worker, find which local certificate has that SKID:

sudo find /var/lib/rancher/k3s/agent -maxdepth 1 -name '*.crt' -print0 |
while IFS= read -r -d '' cert; do
  echo "=== $cert"
  sudo openssl x509 -in "$cert" -noout -subject -issuer -serial \
    -ext subjectKeyIdentifier 2>/dev/null
done

Compare it with:

B0:BB:C0:B3:77:08:6C:1B:04:7B:F1:D0:2D:8F:E9:0E:0F:1C:21:26

If it is a worker's client-kubelet.crt, copy only that public certificate to a temporary path on the server and verify the chain against the authoritative current CA:

openssl verify \
  -CAfile /var/lib/rancher/k3s/server/tls/client-ca.crt \
  /tmp/worker-client-kubelet.crt

Comparing issuer/subject strings is not enough: two different CA keys can have the same subject. openssl verify checks the signature against the actual current key chain.

Also compare the CA fingerprints, rather than only timestamps/names:

# server
sudo openssl x509 -in /var/lib/rancher/k3s/server/tls/client-ca.crt \
  -noout -fingerprint -sha256 -subject

# each worker
sudo openssl x509 -in /var/lib/rancher/k3s/agent/client-ca.crt \
  -noout -fingerprint -sha256 -subject

4. Interpret the result before changing certificates

• If openssl verify succeeds and the tunnel connects, enable K3s debug logging briefly and look for Tunnel server egress proxy dial error; the next likely issue is tunnel session/node-name mapping.
• If verification fails on all workers, the server is now trusting a different client CA from the one that issued their certificates. Do not generate another CA or delete TLS directories. First establish whether a CA rotation/restore occurred at the Jun 17 restart and recover using the documented K3s CA-rotation procedure and current secure agent token.
• If only one worker fails verification, re-enroll/rotate that node's client credentials using the cluster's current token after taking note of its node password and workloads; do not copy another node's certificate.

As a narrow diagnostic only, because you have verified direct server-to-kubelet connectivity, temporarily setting the server to egress-selector-mode: disabled should make kubectl logs use the direct path. If that makes all worker logs work, it conclusively isolates the failure to the agent tunnels. Revert afterwards unless direct control-plane-to-node connectivity is intentionally your permanent security model.

Finally, upgrade from v1.33.3+k3s1 to a currently supported patch before repairing credentials, but do not assume an upgrade alone fixes a CA mismatch.

Primary references:

• K3s control-plane egress selector and agent tunnels: https://docs.k3s.io/networking/basic-network-options#control-plane-egress-selector-configuration
• K3s certificate checking/rotation and CA cautions: https://docs.k3s.io/cli/certificate
• Current tunnel implementation (WebSocket path, TLS config, reconnect log): https://github.com/k3s-io/k3s/blob/master/pkg/agent/tunnel/tunnel.go
• Server tunnel routing to remote kubelets: https://github.com/k3s-io/k3s/blob/master/pkg/daemons/control/tunnel.go

If you post the worker tunnel log plus the openssl verify result, the remaining repair can be made specific instead of guessing. If this diagnosis resolves it, please mark the answer accepted so the default K3s tunnel path is visible to future readers with the same 502.

[brandond]

Thanks, AI.

Several issues with agent tunnels were addressed 8-12 months ago. As I said, @IT-jamf should upgrade first, and see if there is still a problem.