How to mutate with kube-mgmt 0.12.1

[ghost]

Hi, my kube-mgmt is not working well with mutating function. Can someone please help me?

I installed kube-mgmt 0.12.1 in helm with the following values.yaml.

However, the k8s resources (pod) were not mutated.

admissionControllerCA: ""
admissionControllerCert: ""
admissionControllerFailurePolicy: Ignore
admissionControllerKey: ""
admissionControllerKind: MutatingWebhookConfiguration
admissionControllerNamespaceSelector:
  matchExpressions:
  - key: openpolicyagent.org/webhook
    operator: NotIn
    values:
    - ignore
admissionControllerRules:
- apiGroups:
  - '*'
  apiVersions:
  - '*'
  operations:
  - '*'
  resources:
  - '*'
admissionControllerSideEffect: Unknown
authz:
  enabled: true
certManager:
  enabled: false
generateAdmissionControllerCerts: true
hostNetwork:
  enabled: false
image: openpolicyagent/opa
imagePullPolicy: IfNotPresent
imageTag: 0.15.1
livenessProbe:
  httpGet:
    path: /health
    port: 443
    scheme: HTTPS
  initialDelaySeconds: 3
  periodSeconds: 5
logFormat: text
logLevel: debug
mgmt:
  configmapPolicies:
    enabled: true
    namespaces:
    - opa
    - opa-example
    - kube-federation-scheduling-policy
    requireLabel: true
  data:
    enabled: true
  enabled: true
  image: openpolicyagent/kube-mgmt
  imagePullPolicy: IfNotPresent
  imageTag: 0.12.1
  port: 8181
  replicate:
    cluster:
    - v1/namespaces
    namespace:
    - extensions/v1beta1/ingresses
    - v1/pods
    - apps/v1/deployments
    path: kubernetes
opa: false
podDisruptionBudget:
  enabled: false
  minAvailable: 1
port: 443
prometheus:
  enabled: false
rbac:
  create: true
  rules:
    cluster:
    - apiGroups:
      - '*'
      resources:
      - '*'
      verbs:
      - '*'
readinessProbe:
  httpGet:
    path: /health
    port: 443
    scheme: HTTPS
  initialDelaySeconds: 3
  periodSeconds: 5
replicas: 1
sar:
  enabled: true
  image: lachlanevenson/k8s-kubectl
  imagePullPolicy: IfNotPresent
  imageTag: latest
securityContext:
  enabled: false
  runAsNonRoot: true
  runAsUser: 1
serviceAccount:
  create: true
  name: null
serviceMonitor:
  enabled: false
  interval: 15s

I tested with following command and policy.

#!/bin/bash

set -x

kubectl create ns opa-example

kubectl create  -n opa-example configmap policy --from-file ./main.rego
kubectl label -n opa-example configmap policy openpolicyagent.org/policy=rego

kubectl -n opa-example apply -f pod.yaml
kubectl -n opa-example get pod nginx -o json | jq '.metadata.labels'

kubectl -n opa-example annotate pod nginx test-mutation=true
kubectl -n opa-example get pod nginx -o json | jq '.metadata.labels'

package system

main = {
    "apiVersion": "admission.k8s.io/v1beta1",
    "kind": "AdmissionReview",
    "response": {
        "allowed": true,
        "patchType": "JSONPatch",
        "patch": patch_bytes,
    }
} {
    is_create_or_update

    input.request.object.metadata.annotations["test-mutation"]

    patch = [
        {"op": "add", "path": "/metadata/annotations/foo", "value": "bar"},
    ]
    patch_json = json.marshal(patch)
    patch_bytes = base64.encode(patch_json)
}

is_create_or_update { is_create }
is_create_or_update { is_update }
is_create { input.request.operation == "CREATE" }
is_update { input.request.operation == "UPDATE" }

test result.

+ kubectl create ns opa-example
namespace/opa-example created
+ kubectl create -n opa-example configmap policy --from-file ./main.rego
configmap/policy created
+ kubectl label -n opa-example configmap policy openpolicyagent.org/policy=rego
configmap/policy labeled
+ kubectl -n opa-example apply -f pod.yaml
pod/nginx created
+ kubectl -n opa-example get pod nginx -o json
+ jq .metadata.labels
{
  "app": "nginx-pod"
}
+ kubectl -n opa-example annotate pod nginx test-mutation=true
pod/nginx annotated
+ kubectl -n opa-example get pod nginx -o json
+ jq .metadata.labels
{
  "app": "nginx-pod"
}

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx-pod
spec:
  containers:
  - name: mutating-test
    image: nginx:1.14.2
    command: ["sleep", "infinity"]

policy has been recognized by opa

$ k -n opa-example get cm policy -oyaml | k neat
apiVersion: v1
data:
  main.rego: |
    package system

    main = {
        "apiVersion": "admission.k8s.io/v1beta1",
        "kind": "AdmissionReview",
        "response": {
            "allowed": true,
            "patchType": "JSONPatch",
            "patch": patch_bytes,
        }
    } {
        is_create_or_update

        input.request.object.metadata.annotations["test-mutation"]

        patch = [
            {"op": "add", "path": "/metadata/annotations/foo", "value": "bar"},
        ]
        patch_json = json.marshal(patch)
        patch_bytes = base64.encode(patch_json)
    }

    is_create_or_update { is_create }
    is_create_or_update { is_update }
    is_create { input.request.operation == "CREATE" }
    is_update { input.request.operation == "UPDATE" }
kind: ConfigMap
metadata:
  annotations:
    openpolicyagent.org/policy-status: '{"status":"ok"}'
  labels:
    openpolicyagent.org/policy: rego
  name: policy
  namespace: opa-example

[anderseknert]

Hi @t-kusanagi2 👋

I've not really deployed OPA + kube-mgmt via that Helm chart myself, but what does the mutating webhook configuration look like? It would be useful to see that it's actually been deployed, and is configured to send resources of any type (or at least Pod type) to OPA.

Additionally, you'll want to enable decision logging in OPA to see which resources are sent by the kubernetes API server for admission control. This will at least tell you if the problem is a misconfiguration in kubernetes or if there's something in the policy / OPA that makes the mutation not work as intended.

[anderseknert]

@eshepelyuk have you used the Helm chart to deploy OPA for mutation? Should it work like the above?

[eshepelyuk]

No, I don't use OPA as admission controller. Unlikely, I could help with the issue.

[eshepelyuk]

But anyway 0.12.1 is quite old version. It better to try using the latest one - 3.0.0.