It is assumed that configured cluster domain is always "cluster.local"

[adrianlzt]

What happened?

Under some circumstances, Pomerium could configure the destination endpoint with the format BACKEND.NAMESPACE.svc.cluster.local.

This is assuming the cluster domain is always cluster.local, that could be not the case. In Kubspray, for example, you can change that value with cluster_name.

What did you expect to happen?

cluster.local should be a parametrizable value, or, maybe, it could be removed.

If it is removed, BACKEND.NAMESPACE.svc will be tried to be resolved, appending the values in the search domains of the /etc/resolv.conf. But I am not sure if all k8s installs set the needed search values (at least it will need search DOMAIN, like search cluster.local).

The official doc shows an example of the possible /etc/resolv.conf configuration, but in the spec there is no reference about if some search values should be always be present.

I have checked in a Kubespray install and GKE cluster they are present.

Additional context

Related with PR #401

[wasaga]

Currently, endpoints are selected by default, one would have to apply 'use_service_proxy' annotation to the ingress in order to enable that mode.

We may probably have another annotation 'service_proxy_domain', that would default to 'svc.cluster.local'.

[adrianlzt]

In fact I found this error because endpoint selection was not working (#400).

Endpoints will be used also if protocol is https, right?

ingress-controller/pomerium/ingress_to_route.go

Line 247 in 553a780

} else if ic.IsSecureUpstream() && r.TlsServerName == "" {

[wasaga]

Endpoints will be used also if protocol is https, right?

yes

[adrianlzt]

Sorry, I meant that the URL with cluster.local is used if protocol is https:

ingress-controller/pomerium/ingress_to_route.go

Line 248 in 553a780

r.TlsServerName = fmt.Sprintf("%s.%s.svc.cluster.local", backend.Name, ic.Namespace)

[wasaga]

As endpoints are just IP addresses, we need figure out TLS server name, thus it's another place where we try to guess FQDN of a service.

Do you want to update your PR and add an (optional) annotation to customize the service FQDN suffix to be something different from 'cluster.local' (the default) ?

[adrianlzt]

What about removing it completely and relaying in the search of resolv.conf?

[wasaga]

although this is true for Kubspray and GKE, there is no assurance from the spec this is a guaranteed.

To be on the safe side and avoiding breaking someone's installation and do an emergency release, I'd prefer it to be a config option instead.

we may also put it into the global CRD, so that it would become a global option, but that would be more involved as it would require a CRD version bump.

[odoucet]

any update on this ? this actually blocks us from using Pomerium.
I can provide another PR if you point me to what needs to be done. I need this implementation to be compatible with Gateway API.
I can add an annotation "clusterName", but on which resource ? Directly on ingress.pomerium.io/v1.Pomerium ?
If so, How can I access this from pomerium/ingress_to_route.go:getPathServiceHosts() ?

[imax9000]

Just got hit by this. Resolving ${service}.${namespace} works fine in my setup, but the hardcoded svc.cluster.local suffix doesn't match the FQDN.

% kubectl run curl --image=radial/busyboxplus:curl -it --rm
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$ nslookup whoami.default
Server:    10.224.0.10
Address 1: 10.224.0.10 kube-dns.kube-system.svc.cluster.home.arpa

Name:      whoami.default
Address 1: 10.233.217.142 whoami.default.svc.cluster.home.arpa
[ root@curl:/ ]$ nslookup whoami.default.svc
Server:    10.224.0.10
Address 1: 10.224.0.10 kube-dns.kube-system.svc.cluster.home.arpa

Name:      whoami.default.svc
Address 1: 10.233.217.142 whoami.default.svc.cluster.home.arpa
[ root@curl:/ ]$ 

Edit: I have added the following to CoreDNS config as a workaround, but it shouldn't be necessary.

rewrite stop {
   name suffix .cluster.local .cluster.home.arpa answer auto
}