Create a "skip when fail" flag for a unstable convention server

[mhoshi-vm]

Describe the feature request

While exploring the creation of custom convention service, currently the convention controller performs very strict checks and fails the entire podintent when even 1 convention server fails to respond with a PodConventionContext (with Status and AppliedStatus included).

This strictness becomes a barrier for deploying / testing convention servers, as it affects all cartographer workloads in the cluster to be prevented to proceed.

While ideally the convention server should always be stable, it will help developing conventions if we can relax this checking by adding a flag to skip the convention server, if it fails to respond with the correct output.

Is your feature request related to a problem? Please describe

Describe alternatives you've considered

Additional context

[scothis]

a barrier for deploying

If using a tool that can order the creation of resource (like kapp) it can be useful to hold the creation of the ClusterPodConvention until the webhook service is ready.

it will help developing conventions

To take the contrarian position, when developing a convention it's even more important to see errors exposed as hard failures. I want to have confidence that my code is working, or not, and never have ambiguity.

---

If a developer in their own environment really wants this setting, I could see it being available as a cluster wide config value. I don't see introducing this capability on a convention by convention basis, as it's too easy for a dev resource configuration to slip into production. (we don't offer a way to skip TLS verification for the same reason).