How can I disable mutation for managed sidecars

[BitRacer]

I'm using a managed version of service mesh in GKE, Anthos Service Mesh. The managed service mesh injects envoy sidecars and references the containers by tag, not with a sha. The result of mutating the sha on the sidecar is that ASM cannot determine what version is installed and chaos ensues.

Is there a way to disable mutation for a container, or set of containers. for example gcr.io/releases/asm*

I only want to skip this mutation for theses sidecars. Not for the main container, which is signed and delivered with a sha

I can't find any examples of how to set this in the cluster policy or mutating webhook config.

[hectorj2f]

@BitRacer I wouldn't recommend to disable the mutation for a container. In addition to that, I won't recommend it as a security best practice consists on relying on digests instead of tags.

[hectorj2f]

It is not possible today to disable it with the existing source code. Feel free to make any changes to that logic.

[zhuoyang]

It would be nice to be able to disable the mutation, it is more obvious what version of image is running on the cluster
There are container registries that support immutable tags anyway, that could resolve the security concern of using tags.

[Mukuls77]

cosign support verify using image with Tag, as during cosign verify cosign converts tag to digest before doing the verification, so i dont see really the benefit of doing the mutation in policy controller to modify IMAG:TAG -> IMAGE@sha256 as with this in podsepcs we loose the important TAG information, it would be nice if policy controller does not do this mutation and let cosign library does it. do we have an specific reason why we do this in policy controller before calling cosign verify

[senanz]

/assign

[Mukuls77]

Mu suggestion would be to have a flag -disableMutation similar to what we have for disabling TUF (-disableTUF) so if this flag is used than the logic for Mutating TAG to Digest does not happen. so based on the user requirement we can choose to have mutation of the tag to digest or not , currently it is hardcoded in the code.
more over as i commented before cosign library support verify based on TAG so when policy controller invokes cosign verify function let say with TAG as IMAGE reference , cosign will anyway first convert it to digest and then only get the associated signature, so putting this logic in policy controller to always invoke cosign verify function with digest does not bring any additional security. This enhancement will allow user to configure policy controller without mutation in case they need to do so.

[sebastien-prudhomme]

Hi @BitRacer , what do you mean by "chaos ensued". I'm also using Anthos Service Mesh and I see no side effect for now. Maybe when updating Istio?

[senanz]

@hectorj2f - could you please share your thoughts?

[hectorj2f]

As mentioned above, disabling the mutation of sidecars won't follow security best practices.

However, you could try using policy rules which match specific labels so the policy-controller only enforce these policies on specific pods or other type of resources. That could exclude the sidecars pods of istio although I haven't tried myself.

[senanz]

Thank Hector for your comment.
The intent here is not to disable that by default but to have option to do that from the chart as we have disable-tuf for example.
We are facing issue when there is replicaset, we can see that K8S get crazy and it's keep continue creating resources infinite.

[bvarga87]

We've tried this "Match" functinailty that only pods would be in the scope and in a way to pass everything beside that, but seems to be only instrumenting the Validation webhook, not the Mutating one.

We've found as issue with this. Scenario is the following:
A deployment already deployed into a k8s cluster, so mutation not done on it yet.
Policy controller enabled later only. We deployed it later, but it can happen that validation simply turned on later for the given namespace.
Scaling the deployment cause that infinite number of replicasets are generated because of the mutation. As a consequence pods will never be ready after it.
What we've found that replicaset created based on the deployment, replicaset mutated so controller manager found there is difference there, so dropped the replicaset and created a new, which also had a difference and so on...

Is there any issue which would deal with this behavior ?

Btw Connaisseur also supports the different scope for mutation:
https://github.com/sse-secure-systems/connaisseur/releases/tag/v3.4.0

After a quick search it seems kyverno also does the mutation and validation on the pod level:
https://kyverno.io/docs/writing-policies/verify-images/sigstore/#certificate-based-signing-and-verification

[bszollos]

#1105 (comment)
@hectorj2f, what if the mutating webhook would respect the match?

[sjain700]

@hectorj2f Why can't we have something like image:tag@sha ? This way the best practice will not be compromised and tag can be used by ASM for version.

[BitRacer]

What we need here at a minimum. is the ability to whitelist a container so mutation is skipped. A simple regex would do. Let me detail the problem we are experiencing. ASM is managed. We enable the feature and it gets installed and upgraded by a managed service that installs and upgrades ASM for us. ASM containers are tagged with a version. The controller expects to see a certain tag or it assumes the side car is not current, so pods are evicted to be replaced. sidecar injection installs the expected container:tag. Pod runs, sidecar gets mutated and then when the controller makes another pass the tag is no longer the current tag so, you guessed it the pod gets evicted again. While I agree with the desire to stick with a SHA is best it is simply not in our control here. At this point the only way to avoid these needless evictions is not to use sigstore which is unfortunate. A whitelist would need to be explicitly set so I think the security risk is minimal and the whitelist would provide us with more stability when using sigstore.

…

On Sun, Oct 27, 2024, 7:36 AM sjain700 ***@***.***> wrote: @hectorj2f <https://github.com/hectorj2f> Why can't we have something like ***@***.*** ? This way the best practice will not be compromised and tag can be used by ASM for version. — Reply to this email directly, view it on GitHub <#1220 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEBPFV3B6P4LUKN3MZDSY3Z5TM4BAVCNFSM6AAAAABCMMM3QSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZZHE4TQOBUGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>