Multiple Gateway and LoadBalancer type Service in one Knative cluster, Pros and Cons

[bonavadeur]

Multiple Gateway and LoadBalancer type Service in one Knative cluster, Pros and Cons

1. Knative version:

• Kubernetes version v1.28.3
• Knative Serving version 1.15.1
• Istio is installed on Knative, version 1.15.1

2. Description

Hi, I am working with Knative Serving. Although I know that dedicated Load Balancer is expensive and "Knative uses a shared ingress Gateway to serve all incoming traffic within knative service mesh" (cited from Configuring the ingress gateway), but our business require me to do that. I achieved it by modify source code of net-istio-controller and net-istio-webhook. In detail, I made changes:

• comment lines 242 to 244 in istio.go to make net-istio-webhook allows to configure multiple external-gateways
• create another istiooperator.install.istio.io/v1alpha1
• add the below Gateway to config-istio:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-istio
  namespace: knative-serving
data:
  external-gateways: |
      - name: knative-ingress-gateway
        namespace: knative-serving
        service: istio-ingressgateway.istio-system.svc.cluster.local 
      - name: knative-custom-gateway-alpha
        namespace: istio-system-alpha
        service: istio-ingressgateway.istio-system-alpha.svc.cluster.local
_example: >
    ################################
    # ...

After that, I create 2 ksvc named alpha and beta, in two seperate namespaces:

$ kubectl get ksvc -A
NAMESPACE   NAME    URL                              LATESTCREATED   LATESTREADY   READY     REASON
alpha      alpha    http://alpha.alpha.example.com   alpha-00001     alpha-00001   True
beta       beta     http://beta.beta.example.com     beta-00001      beta-00001    Unknown   Uninitialized

Then, I can access to ksvc/alpha through knative-ingress-gateway and access ksvc/beta through knative-custom-gateway-alpha, perfectly.

# make request to ksvc alpha through knative-custom-gateway-alpha
$ curl -HHost:alpha.alpha.example.com 192.168.133.101
Alpha
# make request to ksvc beta through knative-ingress-gateway
$ curl -HHost:beta.beta.example.com 192.168.133.102
Beta
# make request to ksvc beta through  knative-custom-gateway-alpha (no response)
$ curl -HHost:beta.beta.example.com 192.168.133.101
$

3. Question

What are the trade-offs we have to make when making this changes, for example, the first thing I easily recognize is ksvc/beta is marked as Unknown, despite the fact that it worked perfectly. What parts of the Knative Serving will be affected by this changes?

Assume that we knew how to rebuild Knative Serving from source and ready to do it.

Thanks for help.

[skonto]

I think this would require a design doc including how to access alpha from beta as well if needed. Btw we have logic that stamps ingresses as ready that needs to change as well and not covered by the changes above. cc @ReToCode @dprotaso for commenting on this one.