Cilium GatewayAPI Argo Rollout implementation #18
Conversation
xtineskim
changed the title
|
Would it be possible for Isovalent or someone upstream to do a full end-to-end (local) example incorporating the following tools/concepts :
I tried and couldn't get it to work. Thanks! |
|
@xtineskim Many thanks for this contribution. Could you please fix DCO as well? See some instructions here https://github.com/src-d/guide/blob/master/developer-community/fix-DCO.md @jay-jain did you try the example from the PR you mean? Was there a specific error message? I think it would be great if you can provide more info to @xtineskim for debugging purposes... |
|
@kostis-codefresh
I have tried using MetalLB (described here) and the new Cilium L2 Announcement functionality released in Cilium 1.14 to no avail (I'll try and post my notes later today) Regardless, I have gone ahead and tried what is proposed in the PR (using kind) and got stuck pretty early on. @xtineskim's PR methodStep 1: Setup Kind ClusterKind Configuration File Create the cluster: Step 2: Install Gateway APIStep 3: Install CiliumThis is where I fail; when installing Cilium (via Helm chart) with helm upgrade \
--install cilium cilium/cilium \
--version 1.14.0 \
--namespace kube-system \
--reuse-values \
--set kubeProxyReplacement="true" \
--set gatewayAPI.enabled=true
Release "cilium" does not exist. Installing it now.
Error: template: cilium/templates/cilium-agent/daemonset.yaml:637:43: executing "cilium/templates/cilium-agent/daemonset.yaml" at <ne $kubeProxyReplacement "strict">: error calling ne: incompatible types for comparisonI think something is wrong with the Helm chart here: https://github.com/cilium/cilium/blob/v1.14.0/install/kubernetes/cilium/templates/cilium-agent/daemonset.yaml#L637 I could be wrong, but I think the Either way, I'd rather not de-rail this PR since this seems to be an external Cilium Helm chart issue; so feel free to proceed. Thank you both for your contributions to the community; this is a very helpful project and a great starting point! |
Signed-off-by: Christine Kim <[email protected]>
Signed-off-by: Christine Kim <[email protected]>
Signed-off-by: Christine Kim <[email protected]>
xtineskim
force-pushed
the
cilium-gatewayapi
branch
from
1600d24
to
1c5cbab
Compare
|
Okay, I was able to resolve the Helm chart error by using I was able to get through the instructions documented in this PR, however no luck in accessing the Gateway. Step 1: Install kindCreate kind configuration (without default CNI) ---
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: sm
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
networking:
disableDefaultCNI: trueCreate the cluster: kind create cluster --config kind.yamlStep 2: Install Calico CNI to get nodes into ready stateIf we don't have a CNI we can't install Cilium, so we use Calico as a temporary one: Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitializedcurl -sL https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yamlStep 3: Install Gateway APIGW_API_VERSION="v0.7.1"
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/${GW_API_VERSION}/standard-install.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/${GW_API_VERSION}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yamlStep 4: Install Cilium Helm Chart with these flagsif ! helm repo list | grep -q cilium; then
helm repo add cilium https://helm.cilium.io
fi
helm upgrade \
--install cilium cilium/cilium \
--version 1.14.0 \
--namespace kube-system \
--reuse-values \
--set-string kubeProxyReplacement=true \
--set gatewayAPI.enabled=true \
--set ipam.mode=kubernetes \
--set externalIPs.enabled=trueWait till everything is up and ready. You can check status and various configs with: cilium status
cilium config view | grep "enable-gateway-api"
cilium config view | grep "enable-l7-proxy"Step 5: Uninstall Calicokubectl delete -f calico.yamlStep 6: Install and Configure MetalLBInstructions taken from here # Create the address pool
KIND_NET_CIDR=$(docker network inspect kind -f '{{(index .IPAM.Config 0).Subnet}}')
METALLB_IP_START=$(echo ${KIND_NET_CIDR} | sed "[email protected]/[email protected]@")
METALLB_IP_END=$(echo ${KIND_NET_CIDR} | sed "[email protected]/[email protected]@")
METALLB_IP_RANGE="${METALLB_IP_START}-${METALLB_IP_END}"
cat << EOF > metallb_values.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default
namespace: metallb-system
spec:
addresses:
- ${METALLB_IP_RANGE}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default
namespace: metallb-system
spec:
ipAddressPools:
- default
nodeSelectors:
- matchLabels:
kubernetes.io/os: linux
EOF
# Install metallb
helm install \
--namespace metallb-system \
--create-namespace \
--repo https://metallb.github.io/metallb metallb \
metallb \
--version 0.13.10Wait until all pods are up in the kubectl -n metallb-system create -f metallb_values.yamlStep 7: Validate the Gateway Class exists$ kubectl get gatewayclasses.gateway.networking.k8s.io
NAME CONTROLLER ACCEPTED AGE
cilium io.cilium/gateway-controller True 18mStep 8: Deploy Argo Rollouts# Install Argo Rollouts (server)
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yamlStep 9: Deploy the Gateway API Plugincat << EOF > rollouts-plugin-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argo-rollouts-config # must be so name
namespace: argo-rollouts # must be in this namespace
data:
trafficRouterPlugins: |-
- name: "argoproj-labs/gatewayAPI"
location: "https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/download/v0.0.0-rc1/gateway-api-plugin-linux-amd64"
EOF
kubectl apply -f rollouts-plugin-cm.yamlStep 10: Deploy the RBAC components---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gateway-controller-role
namespace: argo-rollouts
rules:
- apiGroups:
- "*"
resources:
- "*"
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gateway-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gateway-controller-role
subjects:
- namespace: argo-rollouts
kind: ServiceAccount
name: argo-rolloutsStep 11: Restart Argo Rollouts DeploymentTo force the CM plugin gets loaded kubectl -n argo-rollouts rollout restart deployment/argo-rolloutsStep 12: Deploy the Gateway, HTTPRoute, Services, and RolloutDeploy the Gateway kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: cilium
spec:
gatewayClassName: cilium
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: AllValidate the gateway $ k get gateway
NAME CLASS ADDRESS PROGRAMMED AGE
cilium cilium 172.18.255.202 True 18m
$ kubectl get gateway cilium -o=jsonpath="{.status.addresses[0].value}"
172.18.255.202Deploy HTTPRoute, Services, and Rollout ---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: argo-rollouts-http-route
spec:
parentRefs:
- kind: Gateway
name: cilium
hostnames:
- "demo.example.com"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: argo-rollouts-stable-service
kind: Service
port: 80
- name: argo-rollouts-canary-service
kind: Service
port: 80
---
apiVersion: v1
kind: Service
metadata:
name: argo-rollouts-canary-service
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: rollouts-demo
---
apiVersion: v1
kind: Service
metadata:
name: argo-rollouts-stable-service
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: rollouts-demo
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollouts-demo
namespace: default
spec:
revisionHistoryLimit: 1
replicas: 10
strategy:
canary:
canaryService: argo-rollouts-canary-service # our created canary service
stableService: argo-rollouts-stable-service # our created stable service
trafficRouting:
plugins:
argoproj-labs/gatewayAPI:
httpRoute: argo-rollouts-http-route # our created httproute
namespace: default
steps:
- setWeight: 30
- pause: {}
- setWeight: 60
- pause: {}
- setWeight: 100
- pause: {}
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollouts-demo
template:
metadata:
labels:
app: rollouts-demo
spec:
containers:
- name: rollouts-demo
image: kostiscodefresh/summer-of-k8s-app:v1
ports:
- name: http
containerPort: 8080
protocol: TCP
resources:
requests:
memory: 32Mi
cpu: 5mStep 13: TestGet Gateway External IP $ GATEWAY="$(kubectl get gateways.gateway.networking.k8s.io cilium -o=jsonpath="{.status.addresses[0].value}")"Access the Gateway $ curl -vvv -H "host: demo.example.com" ${GATEWAY}/call-me
* Trying 172.18.255.202:80...
* connect to 172.18.255.202 port 80 failed: Connection timed out
* Failed to connect to 172.18.255.202 port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to 172.18.255.202 port 80: Connection timed out |
|
@jay-jain you may need to port forward in order to access the IP address of the gateway! |
|
@xtineskim Hmmm I wasn't able to get it to work with a port-forward. I think since it's an external IP it shouldn't be necessary. I found a Cilium issue here which sounds very similar to what I'm experiencing with bare-metal setups. I think once that upstream issue gets resolved, I should have this up and running. |
|
@jay-jain depending on what your env looks like (personally, i use lima vm to run my kind cluster, and have to forward my port (not kubectl port-forward)), but that issue could be it... 🤔 |
|
@xtineskim Ahh gotcha that makes sense. I have no objections to this getting merged now without that upstream issue being resolved, since that issue is pretty specific to bare-metal environments, but I'll leave it up to yourself and @kostis-codefresh . What you have here seems like a really good starting point. I can open a PR later down the road if/when the upstream issue gets resolved with some more specific instructions (if needed). |
Adding instructions for adding the CIlium Gateway API Argo Rollouts