Verify only pods

[dszakallas]

Description

Make policy-controller configurable to only consider pods, not higher level resources.

Use case

We are using a continuous reconciliation (GitOps) solution, Flux, to maintain cluster state. The mutation of the image reference by the mutating admission webhook has undesired effects for us when used in conjunction with Flux:

• Jobs are immutable. It can happen that digest changes for the same tag between reconciliations, causing mutation to happen to the job, which is rejected by kube api server, as it targets an immutable field.
• even for deployments, resolving a digest every sync cycle causes a tight control that forces redeployment whenever the digest changes behind a tag, which we find undesirable.

Workarounds

While the scope of resources to validate can be controlled on the Policy level, this is not sufficient for our use, as mutations still get applied to all recognized resources and this seems to be hard coded. The only workaround I found was to fork the code and remove these.

Related work

Correct me if I am wrong, but I believe this change would not deteriorate the integrity of the cluster, since the pods owned by these unvalidated resources would still go through admission control, and their images will be verified. As an example of this practice, PSA (Pod Security Admission) only operates on the pod resource. Recently, Connaisseur also added a flag to opt out of higher level resource validation.

[senanz]

/assign

[senanz]

@hectorj2f - could you please assign it to me, started to work on code that supporting that.

[0xiso]

@senanz We are also facing the same issue and are searching for solutions. Seems like you are working on solving this problem, but do you have plans to create a PR?

[hectorj2f]

@0xiso @senanz feel free to assign it yourself, but we need to discuss whether we want to go ahead with this change.

[senanz]

@hectorj2f - I have already have the change, need to do some modification and let's discuss that then.
will update you soon.
Thanks for consedring that.

[senanz]

@0xiso - I have the PR need some updates and will raise it again.
@hectorj2f -Could you please assign the ticket on me?

[senanz]

@hectorj2f - Could you please review?
#1687
Let me know of you want to do dsicussion regarding that on slack.