HA Talos on Openstack: chicken-egg problem

[Kirth]

Hi! Talos looks really cool and I'm trying to figure out how to comfortably deploy it on Openstack with a highly available control plane. For this, I'd spin up 3 control nodes and put them behind a load balancer which leads me to a chicken/egg problem:

The load balancer has a different public IP than the nodes themselves, which leads to an error akin to

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for <network internal node IP>, not <external LB IP>"

It's possible to assign this external floating IP straight to one of the control plane nodes and bootstrap it that way. The control nodes form a cluster and I had hoped that they would also exchange information so as to allow communications from the external IP to all nodes but that's not the case. Based on other discussions here, I figured I'd have set the certSANs in cluster.apiServer to include the public IP, attach that IP to the first control-plane node, and bootstrap that node; then move the IP to the load balancer.

I bootstrap by means of:

talosctl --talosconfig talosconfig config endpoint $PUBLIC_IP
talosctl --talosconfig talosconfig config node $PUBLIC_IP
talosctl --talosconfig talosconfig bootstrap

When the IP is pointed straight to my first node talosctl --talosconfig talosconfig health is happy. When I then move the IP to the load balancer, there's issues with the cert being signed by an unknown authority 2/3rd of the time;

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

and the 1/3rd of the time it hits the originally bootstrapped control node which responds but then gets into trouble when it tries to contact the k8s api server over the public IP:

$ talosctl --talosconfig talosconfig health
discovered nodes: ["10.22.0.65" "10.22.0.83" "10.22.0.84"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for no diagnostics: ...
waiting for no diagnostics: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: Get "https://45.135.58.222:6443/api/v1/nodes": EOF
rpc error: code = Unavailable desc = error reading from server: EOF

What am I missing or misunderstanding?

Thank you for your time!
Kirth

[smira]

For Talos API access, use direct IPs of the nodes as endpoints, and their (private) IPs as nodes. talosctl does proper client-side load balancing. Floating IPs are not supported for Talos API access. If you want to remap external IP to some private IP of the VM, include it as .machine.certSANs, but the mapping should be static (required only for controlplanes which are endpoints, other nodes can enjoy private IPs only).

For Kubernetes API, you need to provision a load balancer of some sorts.

Not sure if it answers your question.

[Kirth]

Hi @smira , thank you for your quick response! The solution was indeed to use certSANS, what was blocking me was a mistake in my loadbalancer configuration so that some requests were directed to the Kubernetes API rather than the Talos API.

For those of you wanting to deploy talos on openstack with the control plane behind a HA load balancer, I've written these docs on how to do so for my employers' public docs page: https://docs.leaf.cloud/en/latest/talos/creating-talos-cluster/.

I hope it's ok to share that here, if not please let me know :). Thank you and goodbye!

[smira]

Kirth, thanks for the docs, if you want (and allowed to), you can contribute it to Talos docs as well https://www.talos.dev/v1.9/talos-guides/install/cloud-platforms/