Audit performance

[shin-nien]

Posting a bit of fyi of our experience and I'm trying to find out a bit more about how Audit works for capacity planning reasons.

Our current Kubernetes setup:

• We run about 10k pods across 2000 deployments
• Everything's in dry-run
• We generate about 72k of violations on each run
• We switched to the discovery-client because "audit-from-cache" used >24Gi of memory
• Our controller settings:

  - --operation=audit
  - --operation=status
  - --logtostderr
  - --audit-chunk-size=0
  - --audit-from-cache=false
  - --audit-interval=300
  - --audit-match-kind-only=true

• Config

  sync:
    syncOnly:
    - group: ""
      kind: Namespace
      version: v1
    - group: extensions
      version: v1beta1
      kind: Ingress

With these settings we get reasonable audit times of ~60 minutes. CPU is ~1.75 cores and <1Gi of memory.

But with a bigger config we started to see issues:

    syncOnly:
    - group: ""
      kind: Namespace
      version: v1
    - group: extensions
      version: v1beta1
      kind: Ingress
    - group: policy
      version: v1beta1
      kind: PodDisruptionBudget
    - group: apps
      version: v1
      kind: Deployment
    - group: apps
      version: v1
      kind: ReplicaSet
    - group: apps
      version: v1
      kind: StatefulSet
    - group: apps
      version: v1
      kind: DaemonSet
    - group: batch
      version: v1
      kind: Job
    - group: batch
      version: v1beta1
      kind: CronJob

We observed that if we added more resources to syncOnly then Audit's healthchecks would start to fail intermittently and seemed to be affecting audit overall e.g. you stop seeing violations appear in the logs until much later. We don't have constraints/policies that rely on data.inventory yet so we didn't think there's any reason why adding more to the internal cache would cause Audit issues. Can anyone explain why a difference in sync config would make a difference?

[maxsmythe]

There could be CPU overhead associated with the additional memory management and handling of watches or possibly swapping depending on how the memory is set up. Rego needing to take longer to traverse the data tree could be possible, though hopefully OPA isn't affected by un-traversed data paths.

Memory and CPU usage is expected to be reduced in the near future, once we have finished implementing compiler sharding:

https://docs.google.com/document/d/1ibCxaI-7HyWyDjQNL4iDRMHnrauufIMJ1D6LwIKdlsI/edit#heading=h.sbwdjaeksfbe

[ritazh]

Thanks for raising this @shin-nien! What version of gatekeeper are you using? If you are on v3.7.0+, PTAL

Audit chunk size: set --audit-chunk-size=400 (defaults to 500, 0 = infinite) Lower chunk size can reduce memory consumption of the auditing Pod but can increase the number requests to the Kubernetes API server.

https://open-policy-agent.github.io/gatekeeper/website/docs//gatekeeper/website/docs/audit

I noticed it’s currently set to: --audit-chunk-size=0

Another thing is since you are not using audit from cache, you do not need to sync all the objects to OPA unless you need it for data.inventory in the rego to test things like uniqueness. Sync objects to OPA can increase the memory footprint of gatekeeper. Depending on the number of resources in the cluster you need to sync, you will need to adjust the memory limits given to the audit pod to prevent OOMKilled.
For more details:
https://open-policy-agent.github.io/gatekeeper/website/docs//gatekeeper/website/docs/sync

[shin-nien]

Thanks @ritazh @maxsmythe - we're on 3.6.0 but we're aware of 3.7.0 and the option of swapping out to disk.

Another thing is since you are not using audit from cache, you do not need to sync all the objects to OPA unless you need it for data.inventory in the rego to test things like uniqueness. Sync objects to OPA can increase the memory footprint of gatekeeper.

We noticed this in our testing. We wanted to keep the OPA cache 'full' because we have sub-teams writing policies and we didn't want them to worry about updating the config and releasing gatekeeper; they could write policies independently of us.

In our initial rollout of Gatekeeper we wanted to use audit-from-cache (it seemed to be much faster) so we had a full list of resources in syncOnly but this had the unwanted side-effect of the admission controller using lots of memory too; this makes sense since for uniqueness checks it needs to know the state of the cluster.

We've ended up splitting the audit and controller into separate namespaces so that they could have individual config settings. Giving the admission-controller a minimum config improved performance dramatically. We thought we'd then be able to have a full config for audit and use audit-from-cache. This hasn't been the case and in a roundabout way we've ended up with the same minimal config and switched to using the discovery client.

I think we're happy with what we have for the moment. Though --audit-chunk-size=0 might carry its own risk if one day the payload is too large; we found that chunking into smaller sizes didn't help because the list becomes stale and it looks like the code skips the Kind on error.

I think the lesson learnt for us was that Audit and the controller sharing the same config seems to be incompatible if you have a large cluster and want to audit-from-cache because the controller/webhook struggles to process requests.

[maxsmythe]

That's definitely interesting. I think it also dovetails with the notion of "expensive" constraints, where they take too long to evaluate for a webhook, but may be useful for audit.