SSL Passthrough not working

[ItsKev]

We just started using argo cd and wanted to use it with cert-manager and Let's Encrypt.

Apparently, SSL-Passthrough doesn't seem to work for us. No matter what we try.
Currently our ingress is configured with the following annotations:

cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
nginx.ingress.kubernetes.io/ssl-passthrough: 'true'

We have enabled SSL-Passthrough in our nginx ingress controller as well. Unfortunately, the browser is always telling us certificate not valid (Kubernetes Ingress Controller Fake Certificate)

When port-forwarding the argo-cd service for example the correct certificate.

Anything we are doing wrong?

Ingress config

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  spec:
    rules:
    - http:
        paths:
        - backend:
            serviceName: argo-cd-argocd-server
            servicePort: https
          path: /
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - argocd.network.ovh
      secretName: argocd-secret

[jannfis]

Hi @ItsKev, have you tried without the following annotation;

nginx.ingress.kubernetes.io/backend-protocol: HTTPS

Also, do you have another load balancer or piece in software in front of your Ingress?

[ItsKev]

Thanks for your quick answer. I will try it without the annotation alter.

We are using the Managed Kubernetes Service from OVH and there is a LoadBalancer from OVH in front automatically once we create a load balancer in k8s.

Could this be a problem?

We tried cert-manager with another service (harbor) that does not need SSL-Passthrough and it showed the generated certificate correctly.

[jannfis]

I'm not familiar with OVH, do they put a layer 4 (TCP) or layer 7 (HTTPS) load balancer in front? I'm just asking, because the Nginx Ingress requires the clients (e.g. Layer 7 load balancers in front) to properly use SNI for SSL-passthrough to work correctly. I have seen quite some trouble with e.g. HAproxy load balancers (reverse proxies) being put in front of Nginx-Ingress with SSL-passthrough enabled. However, since OVH seems to create those automatically for services of type LoadBalancer, I'd rather guess they use Layer 4 LB which the SNI issues should not apply to.

If I remember correctly, the backend-protocol and ssl-passthrough annotations are mutually exclusive. The backend-protocol would be used only in reverse-proxy (Layer 7) mode, while ssl-passthrough basically doesn't care about the backend protocol (because it will act a little similar to Layer 4 LB, and only proxies the TCP connection). So this might indeed be the problem after all.

[jannfis]

Another note on the SNI requirement of ssl-passthrough: Your certificate needs to be issued for the very same host name or IP address you are using to access the service, in order for Nginx-Ingress to determine the correct backend. If it doesn't find a matching backend, it will do just as what you described and use the default backend with the default (self-signed) certificate as you described. Also, all clients in the chain need to support SNI.

To illustrate this a little (simplified):

A GET https://example.com HTTP/x will send a SNI request to Nginx-Ingress for example.com. Nginx-Ingress will then perform a TLS handshake to the backend to see whether the certificate matches the SNI. If it does, the connection is handed back to the client to perform a full TLS handshake with the backend. If it doesn't match, Nginx-Ingress will return the client a connection to the default backend.

Update: Also, SNI (and therefore SSL passthrough) might not work with wildcard certificates.

[ItsKev]

Thanks for your help @jannfis

We installed argocd with helm and are now using the integrated ingress which can be used providing the values over the values.yaml. This worked then directly on the first try.

values.yaml

server:
  ingress:
    enabled: true
    annotations:
      "cert-manager.io/cluster-issuer": "letsencrypt-stage"
      "kubernetes.io/ingress.class": "nginx"
      "nginx.ingress.kubernetes.io/ssl-passthrough": "true"
      "nginx.ingress.kubernetes.io/backend-protocol": "HTTPS"
    hosts:
      - example.com
    paths:
      - /
    tls:
      - secretName: argocd-secret
        hosts:
          - example.com
    https: true