Manual sync and diff for ApplicationSet applications

[Agalin]

ApplicationSet is a really useful tool when it comes to deployments to multiple clusters but I find its sync configuration strangely limited.

If for one reason or another (testing, debugging, Kubernetes cluster burning down, whatever) I need to manually modify one of the managed applications I have those options:

• set sync policy to create-only or create-delete,
• set ignoreApplicationDifferences,
• set preservedFields (only for annotations and labels if I understand correctly),
• set whole applicationset controller to dry run.

All those options (except the last one) can be set on per-applicationset basis (assuming policy override is enabled). So far so good. I can configure what's needed then modify the underlying Application object.

So what's missing? Three things really.

• option to view how is specific Application differing from the ApplicationSet (equivalent of the diff view in the Application itself),
• option to list differing Applications (if you set any of those options there is no way to tell which Application has drifted),
• option to re-sync.

Last point is especially interesting because... it's technically doable. With sync policy set to create-only or create-delete you can delete the underlying Application object and controller will recreate it with clean state for you. That's not really a clean solution though.

I know, you might just re-enable automatic updates or remove ignores. Unfortunately that means you need to manually edit the ApplicationSet each time some kind of emergency change to the underlying Application happens. Is this situation rare? Probably, it definitely should be rare. It doesn't make it any less inconvenient, especially when you consider that UI doesn't show you why the field you've just modified has been automatically reverted (there is no automatic link in UI that would show which set manages the app, it only goes the other way around, from set's view to app's).

Yes, you can achieve that using App of Apps instead of ApplicationSet. But that means you lose access to a powerful templating machinery when it comes to cluster selection.

So, what do you think, is it feasible to extend ApplicationSet -> Application view in such way? Or maybe I'm missing some existing solutions after all?

[ChristianCiach]

It's kinda unfortunate that the ApplicationSet controller directly deploys the generated Application manifests to the cluster, isn't it? If there was a way to push the generated Application manifests to Git instead, we could just use an app-of-apps to solve pretty much every issue you mentioned.

I wonder if the upcoming "hydrator" feature could be used as a starting point to achieve something like that (because this will be the first time ArgoCD would natively be able to push stuff to Git).

[ChristianCiach]

I have an idea!

An upcoming version of the ArgoCD CLI will be able to generate a preview (dry-run) of the generated Application manifests:

• feat(cli): add cmd to preview generated apps of appsets (#10895) #16781

So instead of deploying the ApplicationSet to the cluster, we could render the Application manifests outside the cluster, maybe as part of a CI pipeline and then push the rendered manifests back to Git. We could then use a regular app-of-apps to deploy the Applications to the cluster.

Obviously, this means that your CI system would need to connect to the ArgoCD server.

[Agalin]

I think that it could be solved if we go further: (a repo of) ApplicationSets usable as an Application source, just like Helm or Kustomize. Argo would basically need to run the preview generation logic under the hood to render manifests and pass them to the standard flow.

That may even be doable in a plugin with this CLI option. 🤔

[ChristianCiach]

You are right! This should also be doable as a ContentManagementPlugin (as soon as the CLI option becomes available).

This is surely not what the --dry-run option was intended for, but it really opens up some exciting new options.