Documentation/questions - High availability

[absolutejam]

Hey!

First of all, thanks for creating Kilo - it seems like a really useful project for providing wireguard to Kubernetes clusters.

I've been experimenting with Kilo and my aim is to build a multi-cluster mesh with it, but I want to understand the optimal mode of operaion high-availability features.

Obviously there is leader-election within the same location, but it's not clear to me how we can identify the current leader - Is it just by the node that currently contains the kilo.squat.ai/wireguard-ip annotation? Would it potentially be worth exposing this as a (read-only) CRD?

Also, if I wanted to connect 2 clusters, while I understand that only one node is active at a time, would it make sense to add all nodes from both clusters as Peers for redundancy? And additionally, should Kilo/kgctl expose a node's keys even if it's not a leader, for this reason?

Or would it make more sense to have an external floating IP/load-balancer that targets the active node, and if so, that might potentially mean exposing a HTTP endpoint for healthchecks.

Additionally, I noticed there's logic to check for an existing key, or generate one if it is absent (mesh.New). I am considering leveraging a volume to persist keys for my nodes (still one unique key per node), and so that they are deterministic - will this cause any issues? And currently, what is the lifecycle of the key - Is it deterministic in any way (ie. based on node name) or is it potentially a new key every time the DaemonSet is scheduled? (It looks like the latter)

Happy to contribute to the docs & example manifests if it helps out!