HA Zot Loadbalancing in EKS questions

[captainswain]

Hello everyone,

I'm currently exploring the deployment of an HA Zot registry on Amazon EKS using S3 and have some questions regarding the best practices and necessary components for this setup.

From the documentation and various resources, I understand that Zot supports clustering with stateless Zot instances/replicas, a shared remote backend storage (S3 API-compatible), and a load balancer for high availability. My main concerns are about the load-balancing approach and how to efficiently manage traffic within the EKS environment.

Questions:

• HAProxy Requirement: Is HAProxy a mandatory component for load balancing in this HA setup? The Zot documentation mentions HAProxy, but I'm wondering if it's strictly required for baremetal or if there are alternative approaches within EKS.
• Ingress Controller Usage: In the external traffic scenario is Nginx-Ingress enough in terms of loadbalancing or should I be pointing the ingress to HAproxy or similar?
• Internal Load Balancing: Most importantly, I want to ensure efficient load balancing without causing traffic to leave the cluster and re-enter (to avoid unnecessary bandwidth costs). How can this be achieved in EKS? Are there specific configurations or best practices to follow to prevent this "hairpinning" of traffic? Could I just use CoreDNS?

I am particularly interested in understanding the trade-offs and best practices for these components in an HA Zot deployment on EKS. Any insights, experiences, or recommendations would be greatly appreciated.

Thank you for your time and help!

[rchincha]

@captainswain

clustering with stateless Zot instances/replicas

This is not really HA but load-balancing. HA is provided by the backend storage. Reverse proxy rules just make sure instances handle parts of the entire registry/repo namespace.

HAProxy Requirement

Any L7 LB is fine, L7 because URIs can be specified and routed to various instances.

Internal Load Balancing

The scenario is not clear. Clients read from or write to zot via the LB which in turn writes to S3. Where is the hairpinning happening and how?

[captainswain]

@rchincha

Thanks for the response! I updated the discussion to better represent what we are discussing. I appreciate you highlighting the distinction between load balancing and HA, as well as the importance of L7 load balancing for URI routing. I'd like to provide some additional details to clarify our setup and the specific concerns I have:

Ingress-Nginx and Load Balancing:
We are currently using ingress-nginx, which creates a Classic Load Balancer in AWS. Given your point about the suitability of any L7 load balancer, it seems that ingress-nginx should be adequate for our needs, as it allows for effective URI-based routing. This setup should handle the load balancing aspect for external traffic to our Zot registry effectively.

Concern about Hairpinning:
The specific scenario I'm concerned about regarding hairpinning involves the way internal cluster components access the Zot registry. Our registry would be accessed via an external URL (e.g., zot.mytld.com) managed by ingress-nginx. When internal components, like CI/CD tools or other Kubernetes pods, need to access the registry, they would use this external URL. My concern is that this could lead the traffic to exit the EKS cluster to reach the public IP of the ingress/LB, and then re-enter the cluster, resulting in unnecessary egress costs. To mitigate this, would it be advisable to configure internal DNS (possibly through CoreDNS) to resolve zot.mytld.com to internal addresses, ensuring that internal requests to the Zot registry do not leave the internal AWS network? Are there other best practices or configurations within EKS that can help in this situation or is it a non-issue?

Your insights into these matters will be greatly appreciated and apologies for my lack of understanding.

[rchincha]

external clients <-> zot.mytld.com (k8s/ingress) <-> zot.svc.cluster.local (k8 svc) <-> AWS S3

Internal clients/pods can just reach zot.svc.cluster.local