AWS Internal NLB with AWS EKS Cluster in Private Subnet of VPC - 307 Too many redirects error on ArgoCD UI

[k-lombard]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I'm getting a 307 too many redirects error when trying to access the URL associated with my ArgoCD UI deployment. I have an internal AWS NLB trying to forward https traffic on port 443 to my AWS EKS node groups. The target's health checks of the node group instances is healthy. I am basically trying to expose an internal, secure ArgoCD UI at a specific staging URL for internal monitoring of our AWS EKS cluster. This AWS EKS cluster is inside the private subnet of our VPC.

I can telnet the internal IP of my AWS NLB on port 443 successfully. I can also setup a local port forwarding with kubectl port-forward svc/argocd-server -n argocd 8080:443 and access the ArgoCD UI at localhost:8080

argocd-cm.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.staging.redacted.com
  server.insecure: 'true'
  server.x-forwarded-proto: 'true'
  server.x-forwarded-host: 'true'
  server.x-forwarded-prefix: 'true'
  server.tls.enabled: 'false'
  server.enable.hsts: 'false'
  # server.baseHref: '/argocd'

argocd-network-policy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-server-network-policy
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: argocd-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 10.0.0.0/16 # VPC CIDR
        - ipBlock:
            cidr: redacted/32 # Specific IP allowed in ALB security group
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - protocol: TCP
          port: 80
        - protocol: TCP
          port: 443
        - protocol: TCP
          port: 8080
        - protocol: TCP
          port: 30007

argocd-server-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-target-node-labels: kubernetes.io/os=linux
    service.beta.kubernetes.io/aws-load-balancer-target-type: ip
    # service.beta.kubernetes.io/aws-load-balancer-ssl-cert: 'cert-arn-redacted'
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-name: argocd-internal-nlb
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    # service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443'
    # service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: 'ELBSecurityPolicy-TLS-1-2-2017-01'
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "{'Environment':'staging'}"
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8080
      nodePort: 30007 # Choose an available port in the 30000-32767 range
    - name: http
      port: 80
      targetPort: 8080
    - name: https
      port: 443
      targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server

relevant terraform info:

resource "aws_lb" "argocd_internal_nlb" {
  name                       = "argocd-internal-nlb"
  internal                   = true
  load_balancer_type         = "network"
  subnets                    = [aws_subnet.private_eks_a.id, aws_subnet.private_eks_b.id]
  security_groups            = [aws_security_group.nlb_sg.id]
  enable_deletion_protection = false
}

resource "aws_lb_listener" "argocd_https" {
  load_balancer_arn = aws_lb.argocd_internal_nlb.arn
  port              = "443"
  protocol          = "TLS"

  ssl_policy      = "ELBSecurityPolicy-TLS-1-2-2017-01"
  certificate_arn = "redacted-arn"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.argocd_tg_nlb.arn
  }
}

resource "aws_security_group_rule" "allow_nlb_to_eks" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  security_group_id = aws_security_group.eks_cluster.id
  cidr_blocks       = [aws_subnet.private_eks_a.cidr_block, aws_subnet.private_eks_b.cidr_block]
}

resource "aws_lb_target_group" "argocd_tg_nlb" {
  name        = "argocd-tg-nlb"
  port        = 30007
  protocol    = "TCP"
  vpc_id      = aws_vpc.main.id
  target_type = "instance"

  health_check {
    path                = "/healthz"
    port                = 30007
    protocol            = "HTTP"
    interval            = 30
    healthy_threshold   = 3
    unhealthy_threshold = 3
  }
}

resource "aws_security_group_rule" "allow_nlb_to_eks_nodeport" {
  type              = "ingress"
  from_port         = 30007
  to_port           = 30007
  protocol          = "tcp"
  security_group_id = aws_security_group.eks_cluster.id
  cidr_blocks       = [aws_subnet.private_eks_a.cidr_block, aws_subnet.private_eks_b.cidr_block]
}


resource "aws_security_group" "nlb_sg" {
  name        = "nlb-security-group"
  description = "Security group for NLB"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "HTTPS from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  ingress {
    description = "HTTPS from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["18.191.176.92/32"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "nlb-security-group"
  }
}

resource "aws_acm_certificate" "argocd_cert" {
  domain_name               = "argocd.${var.custom_domain}"
  validation_method         = "DNS"
  subject_alternative_names = ["www.argocd.${var.custom_domain}"]

  lifecycle {
    create_before_destroy = true
  }

  tags = {
    Environment = var.environment
  }
}

resource "aws_route53_record" "argocd_cert_validation" {
  for_each = {
    for dvo in aws_acm_certificate.argocd_cert.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = aws_route53_zone.redacted.zone_id
}

resource "aws_acm_certificate_validation" "argocd_cert" {
  certificate_arn         = aws_acm_certificate.argocd_cert.arn
  validation_record_fqdns = [for record in aws_route53_record.argocd_cert_validation : record.fqdn]
}

resource "aws_route53_record" "argocd" {
  zone_id = aws_route53_zone.redacted.zone_id
  name    = "argocd.${var.custom_domain}"
  type    = "A"

  alias {
    name                   = var.argocd_aws_lb_dns_name
    zone_id                = var.argocd_aws_lb_zone_id
    evaluate_target_health = true
  }
}

resource "aws_route53_record" "argocd_www" {
  zone_id = aws_route53_zone.redacted.zone_id
  name    = "www.argocd.${var.custom_domain}"
  type    = "A"

  alias {
    name                   = var.argocd_aws_lb_dns_name
    zone_id                = var.argocd_aws_lb_zone_id
    evaluate_target_health = true
  }
}

To Reproduce

Expected behavior

Should be able to access the argocd ui internally without getting a 307 too many redirects error.
Screenshots

Version

argocd version
argocd: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T15:33:41Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.23.0
  Compiler: gc
  Platform: darwin/arm64

Logs

kubectl logs -l app.kubernetes.io/name=argocd-server -n argocd
time="2024-09-11T15:27:30Z" level=info msg="Enabled application namespace patterns: argocd"
time="2024-09-11T15:27:30Z" level=info msg="0xc000d7d080 subscribed to settings updates"
time="2024-09-11T15:27:30Z" level=info msg="Starting rbac config informer"
time="2024-09-11T15:27:30Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
time="2024-09-11T15:27:57Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-secret"
time="2024-09-11T15:27:57Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-cm"
time="2024-09-11T15:30:57Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-secret"
time="2024-09-11T15:30:57Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-cm"
time="2024-09-11T15:33:57Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-secret"
time="2024-09-11T15:33:57Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-cm"

Curl:
curl -v https://argocd.staging.redacted.com

• Trying 10.0.7.216:443...
• Connected to argocd.staging.redacted.com (10.0.7.216) port 443
• ALPN: curl offers h2,http/1.1
• (304) (OUT), TLS handshake, Client hello (1):
• CAfile: /etc/ssl/cert.pem
• CApath: none
• (304) (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
• ALPN: server did not agree on a protocol. Uses default.
• Server certificate:
• subject: CN=argocd.staging.redacted.com
• start date: Sep 9 00:00:00 2024 GMT
• expire date: Oct 8 23:59:59 2025 GMT
• subjectAltName: host "argocd.staging.redacted.com" matched cert's "argocd.staging.redacted.com"
• issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M03
• SSL certificate verify ok.
• using HTTP/1.x

GET / HTTP/1.1
Host: argocd.staging.redacted.com
User-Agent: curl/8.4.0
Accept: /

< HTTP/1.1 307 Temporary Redirect
< Content-Type: text/html; charset=utf-8
< Location: https://argocd.staging.redacted.com/
< Date: Wed, 11 Sep 2024 15:34:34 GMT
< Content-Length: 73
<
Temporary Redirect.

• Connection #0 to host argocd.staging.redacted.com left intact