fix(manifests): fix full CRDs and update tests to use them. Fixes #8532 #14044

MasonM · 2025-01-02T00:49:13Z

Fixes #8532. Partial fix for #8190

Motivation

The full CRDs at manifests/base/crds/full have many errors that make them only usable in editors (#11266 (comment)). They used to be installable in a cluster, but now that's no longer possible:

$ kubectl apply --server-side -k manifests/base/crds/full/
customresourcedefinition.apiextensions.k8s.io/workflowartifactgctasks.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io serverside-applied
Error from server (Invalid): CustomResourceDefinition.apiextensions.k8s.io "clusterworkflowtemplates.argoproj.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[templateDefaults].properties[dag].properties[tasks].items.properties[inline].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[templateDefaults].properties[steps].items.items: Required value: must be specified, spec.validation.openAPIV3Schema.properties[spec].properties[templates].items.properties[dag].properties[tasks].items.properties[inline].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[templates].items.properties[steps].items.items: Required value: must be specified]
Error from server (Invalid): CustomResourceDefinition.apiextensions.k8s.io "cronworkflows.argoproj.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[workflowSpec].properties[templateDefaults].properties[dag].properties[tasks].items.properties[inline].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[workflowSpec].properties[templateDefaults].properties[steps].items.items: Required value: must be specified, spec.validation.openAPIV3Schema.properties[spec].properties[workflowSpec].properties[templates].items.properties[dag].properties[tasks].items.properties[inline].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[workflowSpec].properties[templates].items.properties[steps].items.items: Required value: must be specified]
...

Having working full CRDs would enable many useful features, such as kubectl explain output (#8190), validating admission policies (#14045), and integration with tools like cdk8s (#8532).

Modifications

This PR looks intimidating, but 99% of the changes are in manifests/base/crds/full, which is not used in any of the release manifests. I tried to avoid changing the release manifests or API as much as possible, so there shouldn't be any backwards-compatibility concerns.

As explained at #13754 (comment), there were two issues that prevented us from using the full CRDs everywhere:

Kubernetes size limits when using client-side apply.
Wrong validation information due to kubebuilder limitations.

For the first issue, I verified that this is no longer an issue when using server-side apply, so I updated the make install target to use it. Since kubectl apply --server-side isn't compatible with kubectl apply --prune, I had to switch to apply sets, which is intended to replace --prune, and seems to work well.

For the second issue, I went through and added workarounds to hack/manifests/crds.go for all the kubebuilder issues I could find. Since the E2E test suite now uses the full CRDs, it should tell us if there's anything I missed.

I didn't update the release manifests to use the full CRDs, since that's a big change and we probably should wait awhile to make sure there aren't any unexpected issues. In the meantime, users can easily opt into the full CRDs if they want.

I did some minor refactoring of the manifests under manifests/ to use Kustomize Components so that we can share code with the the manifests under test/e2e/manifests without duplication. See #14001 for more details on this approach. EDIT: this was split off to #14048 to make this PR easier to review.

Verification

Ran tests locally. Also, I verified the documentation changes using make docs-serve.

MasonM · 2025-01-02T00:57:30Z

test/e2e/manifests/components/base/kustomization.yaml

-labels:
- includeSelectors: true
-  pairs:
-    app.kubernetes.io/part-of: argo


FYI: This label was only needed when using the old --prune method. When using apply sets, Kubernetes automatically generates the appropriate label.

MasonM · 2025-01-02T00:59:22Z

test/e2e/malformed_resources_test.go

-			assert.Equal(t, "malformed", metadata.Name)
-			assert.Equal(t, wfv1.WorkflowFailed, status.Phase)
-		})
+	s.Given().Exec("kubectl", []string{"apply", "-f", "testdata/malformed/malformed-workflow.yaml"}, fixtures.ErrorOutput("unknown field \"spec.arguments.parameters.someParam\""))


I had to substantially change the tests in this file because malformed workflows will now be rejected at the API level instead of at runtime. We could test the runtime behavior of malformed workflows by adding a new profile that uses the minimal CRDs and having test-executor use that, but that seems overkill.

The full CRDs at `manifests/base/crds/full` are only intended for usage in editors (argoproj#11266 (comment)) and are unusable otherwise. Trying to install them will cause spurious validation errors with nearly every workflow. Having the full CRDs would enable many useful features, such as `kubectl explain` output (argoproj#8190), [validating admission policies](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) (argoproj#13503), and integration with tools like cdk8s (argoproj#8532). As explained at argoproj#13754 (comment), there's two issues that prevent us from using the full CRDs everywhere: 1. [Kubernetes size limits](kubernetes/kubernetes#82292) when using client-side apply. This is not a problem when using . 2. Wrong validation information due to kubebuilder limitations. For the first issue, I verified that this is no longer an issue when using [server-side apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/), so I updated the `make install` target to use it. Since `kubectl apply --server-side` isn't compatible with `kubectl apply --prune`, I had to switch to use [apply sets](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune), which is intended to replace `--prune`, and seems to work well. For the second issue, I went through and added workarounds to `hack/manifests/crds.go` for all the kubebuilder issues I could find. Since the E2E test suite now uses the full CRDs, it should tell us if there's anything I missed. I didn't update the release manifests to use the full CRDs, since that's a big change and we probably should wait awhile to make sure there aren't any unexpected issues. Users can easily opt into the full CRDs if they want. Signed-off-by: Mason Malone <[email protected]>

MasonM · 2025-01-02T01:25:36Z

docs/installation.md

+
+The official release manifests come with stripped-down CRDs that omit validation information.
+This is a workaround for [Kubernetes size limitations](https://github.com/kubernetes/kubernetes/issues/82292) when using client-side apply.
+As of version 3.7, the full CRDs can be installed using [server-side apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) via the following command:


If this doesn't go out as part of 3.7, I'll remove this before it's merged

This was split off from argoproj#14044 to make it easier to review. This makes a few minor improvements to the build to support the full CRD fixes: 1. Update devcontainer to install k3s 1.29.10, since that's what we use in CI: https://github.com/argoproj/argo-workflows/blob/ef41f83f801a6ff48c87f882a6b75d0e37529134/.github/workflows/ci-build.yaml#L263 1.27 is EOL and doesn't support things like validation rules in CRDs. 2. Update `make install` to use [server-side apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/), which is meant to replace client-side apply and isn't affected by size limitations for the CRDs. Since `kubectl apply --server-side` isn't compatible with `kubectl apply --prune`, I had to switch to [apply sets](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune), which is intended to replace allow list pruning, and seems to work just as wlel. 3. Minor refactoring of the manifests under `manifests/` to use [Kustomize Components](https://kubectl.docs.kubernetes.io/guides/config_management/components/) so that we can share code with the the manifests under `test/e2e/manifests` without duplication. See argoproj#14001 for more details on this approach. Signed-off-by: Mason Malone <[email protected]>

MasonM · 2025-01-03T02:09:35Z

I split off the minor build refactoring (e.g. switching to server-side apply) to a separate PR to make this one easier to review: #14048

MasonM commented Jan 2, 2025

View reviewed changes

MasonM force-pushed the full-crds2 branch from 0d64c1d to 57ba14a Compare January 2, 2025 01:23

MasonM commented Jan 2, 2025

View reviewed changes

MasonM added type/bug area/manifests labels Jan 2, 2025

MasonM marked this pull request as ready for review January 2, 2025 01:53

MasonM requested review from Joibel, terrytangyuan and sarabala1979 as code owners January 2, 2025 01:53

This was referenced Jan 3, 2025

build: minor upgrades/refactoring to support full CRDs #14048

Open

argo cron should report when CronWorkflow names exceed the length limit #6781

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(manifests): fix full CRDs and update tests to use them. Fixes #8532 #14044

fix(manifests): fix full CRDs and update tests to use them. Fixes #8532 #14044

MasonM commented Jan 2, 2025 •

edited

Loading

MasonM Jan 2, 2025

MasonM Jan 2, 2025 •

edited

Loading

MasonM Jan 2, 2025

MasonM commented Jan 3, 2025

fix(manifests): fix full CRDs and update tests to use them. Fixes #8532 #14044

Are you sure you want to change the base?

fix(manifests): fix full CRDs and update tests to use them. Fixes #8532 #14044

Conversation

MasonM commented Jan 2, 2025 • edited Loading

Motivation

Modifications

Verification

MasonM Jan 2, 2025

Choose a reason for hiding this comment

MasonM Jan 2, 2025 • edited Loading

Choose a reason for hiding this comment

MasonM Jan 2, 2025

Choose a reason for hiding this comment

MasonM commented Jan 3, 2025

MasonM commented Jan 2, 2025 •

edited

Loading

MasonM Jan 2, 2025 •

edited

Loading