Masking sensitive data in decision logs is not working in conjunction with remote policy bundle from GCS

[kdonthireddy]

Short Description

I've setup a Kubernetes cluster and configured Istio in my local as per pre-requisites mentioned here: https://www.openpolicyagent.org/docs/latest/envoy-tutorial-istio/. I was able to successfully complete the tutorial, and then continued to explore more advanced OPA concepts like remote (GCS) policy bundles and masking sensitive data in decision logs using same setup. I've even tested my masking logic using opa cli, however, noticed that masking is not working when my authorization policy bundle is being fetched from GCS.

Steps To Reproduce

# Configuration to bootstrap OPA-Istio sidecars
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-config
  namespace: bookinfo
data:
  config.yaml: |
    plugins:
      envoy_ext_authz_grpc:
        addr: :9191
        path: istio/authz/allow
    decision_logs:
      console: true
    services:
      gcs:
        url: https://storage.googleapis.com/storage/v1/b/policy-bundle/o
        credentials:
          gcp_metadata:
            scopes:
            - https://www.googleapis.com/auth/devstorage.read_only
    bundles:
      authz:
        service: gcs
        resource: 'opa-policy.tar.gz?alt=media'
---
# Masking policy to enforce on decision logs from OPA-Istio sidecars
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-mask
  namespace: bookinfo
data:
  mask.rego: |
    package system.log

    import rego.v1

    mask contains "/input/password"

    mask contains "/input/attributes/request/http/headers/authorization"
---
# OPA Admission Control Policy for injecting OPA-Istio sidecars
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-inject-policy
  namespace: opa-istio
data:
  inject.rego: |
    package istio

    uid := input.request.uid

    inject = {
      "apiVersion": "admission.k8s.io/v1",
      "kind": "AdmissionReview",
      "response": {
        "allowed": true,
        "uid": uid,
        "patchType": "JSONPatch",
        "patch": base64.encode(json.marshal(patch)),
      },
    }

    patch = [{
      "op": "add",
      "path": "/spec/containers/-",
      "value": opa_container,
    }, {
      "op": "add",
      "path": "/spec/volumes/-",
      "value": opa_config_volume,
    }, {
      "op": "add",
      "path": "/spec/volumes/-",
      "value": opa_mask_volume,
    }]

    opa_container = {
      "image": "openpolicyagent/opa:latest-istio",
      "name": "opa-istio",
      "args": [
        "run",
        "--server",
        "--config-file=/config/config.yaml",
        "--addr=localhost:8181",
        "--diagnostic-addr=0.0.0.0:8282",
        "/mask/mask.rego",
      ],
      "volumeMounts": [{
        "mountPath": "/config",
        "name": "opa-istio-config",
      }, {
        "mountPath": "/mask",
        "name": "opa-mask",
      }],
      "readinessProbe": {
        "httpGet": {
          "path": "/health?plugins",
          "port": 8282,
        },
      },
      "livenessProbe": {
        "httpGet": {
          "path": "/health?plugins",
          "port": 8282,
        },
      }
    }

    opa_config_volume = {
      "name": "opa-istio-config",
      "configMap": {"name": "opa-config"},
    }

    opa_mask_volume = {
        "name": "opa-mask",
        "configMap": {"name": "opa-mask"},
    }
---
# OPA Admission Controller deployment for injecting OPA-Istio sidecars
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opa-admission-controller
  namespace: opa-istio
  labels:
    app: opa-admission-controller  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa-admission-controller
  template:
    metadata:
      name: opa-admission-controller
      labels:
        app: opa-admission-controller
    spec:
      containers:
        - image: openpolicyagent/opa:latest
          name: opa
          ports:
          - containerPort: 8443
          args:
          - "run"
          - "--server"
          - "--tls-cert-file=/certs/tls.crt"
          - "--tls-private-key-file=/certs/tls.key"
          - "--addr=0.0.0.0:8443"
          - "/policies/inject.rego"
          livenessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTPS
              port: 8443
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTPS
              port: 8443
            initialDelaySeconds: 5
            periodSeconds: 5
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: server-cert
            - readOnly: true
              mountPath: /policies
              name: inject-policy
      volumes:
        - name: inject-policy
          configMap:
            name: opa-inject-policy
        - name: server-cert
          secret:
            secretName: opa-admission-controller-secret

Expected Behavior

I was expecting that masking policy (through a ConfigMap) would work alongside the authorization policy bundle being fetched from remote. While authorization policy enforcement is working fine, sensitive data in decision logs is not masked despite no errors reported in server logs.

[ashutosh-narkar]

The policy you're providing on the command-line via ConfigMap won't get loaded into OPA alongside the bundle. I need to check the code but iirc OPA just won't load it with a config provided. You could define the policy in the bundle itself or use multiple bundles.

[charlieegan3]

As Ash says, OPA will not load the Rego policy from the mounted dir when the config is specified. You might want to try running a local sidecar that serves that policy over the localhost to OPA if you need to masking policy to be defined in k8s state.