Evaluate admission controller design for Kubernetes verification of models

[font]

Description

It would be great to have an approach to verify models within Kubernetes clusters as part of an admission controller similar to Sigstore's Policy Controller. Currently we have a PoC implementation that relies on a mutating webhook to mutate pod specs to add an initcontainer that handles verifying the model using a containerized version of the verify CLI from this repo (see sigstore/TSC#57 for more details).

We can view this mutating webhook approach as a more asynchronous approach to verification of models as the pod consuming the model will fail to start if the initcontainer added by the mutating webhook fails because model verification failed.

Sigstore's Policy Controller operates in a more synchronous approach by verifying the OCI image is signed upon receiving the admission request and fails the admission immediately if verification fails. Although Sigstore's Policy admission controller is more synchronous, the asynchronous approach using a mutating webhook may be the best we can do right now. In order to be more synchronous and to potentially integrate with Sigstore's Policy Controller (if that becomes desirable), we would need to consider the following:

1. A Golang implementation of this model-transparency library could simplify either controller solutions by allowing the controller to call directly into a Golang library instead of having to OS exec into a Python-based verify CLI. See Golang implementation of this library #414.
2. Add a Storage Access Service for synchronous model access for a Kubernetes admission controller.
3. Using OCI standard for packaging and deployment of ML models within cloud native environments like Kubernetes. See Considerations for using the OCI spec for signing and verifying models #434 for details.