Steward

Kubernetes shared cluster provisioning for knative

The design is that this is a webserver which runs on the cluster with a privileged service account. When a user arrives it will serve a page with Google Login available. Once the user has logged in with Google login, it will provision a Namespace, ResourceQuota, LimitRanger, Broker and RoleBindings for that user, and then serve the user a custom KUBECONFIG file that includes the cluster information (including server keys), namespace, and GCP auth commandline. The user

Setup

... this is longer than I'd like, but should allow for complete demos of Knative on a shared cluster where the participants can just walk up and sign in with Google and then get started with kn commands.

On the cluster administrator side, you'll need to set up a Kubernetes cluster which support the Google auth (I've only tested GKE, not sure what's required to get the OAuth to work elsewhere), and then do the following:

1. Have a domain that you can use for serving both the user applications and the provisioning application. You'll need to be able to set some DNS records on that domain later. In this description, we use kube.demo.app as the domain.

2. Create a Google Project via http://console.cloud.google.com/ and create an OAuth2 Client ID here: https://console.cloud.google.com/apis/credentials?project=kubecon-knative-2019

   Follow the instructions for web signin and backend signin for setting up the authorization credentials. In particular, you'll

3. Install Istio: https://knative.dev/docs/install/installing-istio/

4. Install the serving operator: https://github.com/knative/serving-operator

   kubectl apply -f https://github.com/knative/serving-operator/releases/download/v0.10.0/serving-operator.yaml

5. Install eventing and monitoring (not using the operator): https://knative.dev/docs/install/knative-with-any-k8s/#installing-knative

   kubectl apply --selector knative.dev/crd-install=true \
   --filename https://github.com/knative/eventing/releases/download/v0.10.0/release.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.10.0/monitoring.yaml

   And reapply if needed due to CRD races.

6. Configure a wildcard domain (*.kube.demo.app IN A $CLUSTER_IP) pointing at your Istio istio-ingressgateway in the istio-system namespace.

7. Patch the config-domain ConfigMap with the new DNS address. Note that the serving operator may reset the config-domain, so you'll want to keep this command handy. Symptoms of the reset are that all KServices stop working at once, and this command reports "patched" rather than "patched (no change)".

   kubectl patch configmap config-domain -n knative-serving --patch "{\"data\": {\"example.com\": null, \"kube.majordemo.com\": \"\"}}"

   (The terrible quoting allows the command to work from cmd.exe in a pinch.)

8. Create the following ClusterRoles needed for Steward and the created resources:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: provision
   rules:
     - apiGroups:
         - ''
       resources:
         - namespaces
         - resourcequotas
         - limitranges
       verbs:
         - '*'
     - apiGroups:
         - rbac.authorization.k8s.io
       resources:
         - rolebindings
       verbs:
         - '*'
     - apiGroups:
         - rbac.authorization.k8s.io
       resourceNames:
         - workshop-user
     - event-viewer
       resources:
         - clusterroles
       verbs:
         - bind
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: workshop-user
   rules:
     - apiGroups:
         - serving.knative.dev
       resources:
         - services
       verbs:
         - '*'
     - apiGroups:
         - serving.knative.dev
       resources:
         - '*'
       verbs:
         - get
         - list
         - watch
     - apiGroups:
         - ''
       resources:
         - configmaps
       verbs:
         - '*'
     - apiGroups:
         - ''
       resources:
         - namespaces
       verbs:
         - get
         - list
         - watch
     - apiGroups:
         - ''
       resources:
         - pods
         - pods/list
       verbs:
         - get
         - list
         - watch
         - delete
     - apiGroups:
         - eventing.knative.dev
       resources:
         - triggers
       verbs:
         - '*'
     - apiGroups:
         - sources.eventing.knative.dev
       resources:
         - apiserversources
       verbs:
         - '*'
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: event-viewer
   rules:
     - apiGroups:
         - ''
       resources:
         - events
       verbs:
         - get
         - watch
         - list

9. Create the following ClusterRoleBinding for launching Steward as the provision service in namespace default with the default service account:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: provisioner
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: provision
   subjects:
     - kind: ServiceAccount
       name: default
       namespace: default

10. Build the steward application into an image. I use gcloud build like so:

    gcloud builds submit --project $PROJECT_ID --tag gcr.io/$PROJECT_ID/steward:live-$COUNTER

11. Deploy the Steward application in the default namespace of the cluster. Note that you'll need the client ID (the part before .apps.googleusercontent.com) from step 2 and to know the address of your Kubernetes master.

    kn service create provision --image gcr.io/$PROJECT_ID/steward:live-$COUNTER --env CLIENT_ID=$CLIENT_ID --env MASTER_ADDRESS=https://1.2.3.4

12. If you want to expose Grafana and Kibana from the cluster as well, you'll need to add a VirtualService like the following:

    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: show-grafana
      namespace: default
    spec:
      # This is the gateway shared in knative service mesh.
      gateways:
        - knative-ingress-gateway.knative-serving.svc.cluster.local
      # Set host to the domain name that you own.
      hosts:
        - dashboard.kube.majordemo.com
      http:
        - match:
            - uri:
                exact: '/logging'
          rewrite:
            uri: '/'
          route:
            - destination:
                host: kibana-logging.knative-monitoring.svc.cluster.local
                port:
                  number: 5601
        - match:
            - uri:
                prefix: '/logging/'
          rewrite:
            # Rewrite the URI header to remove the prefix for Kibana. See
            # https://stackoverflow.com/questions/36266776/kibana-server-basepath-results-in-404
            # for details.
            uri: '/'
          route:
            - destination:
                host: kibana-logging.knative-monitoring.svc.cluster.local
                port:
                number: 5601
        - match:
            - uri:
                prefix: '/'
          route:
            - destination:
                host: grafana.knative-monitoring.svc.cluster.local
                port:
                  number: 30802

    Note that you'll also need to update the default Kibana installation to change the SERVER_BASEPATH environment variable in the Deployment to /logging (no trailing "/").

End result

You should now be able to walk through the steps in this tutorial.