Bootstrapping: ApplicationSets vs App-of-apps vs Kustomize

[boedy]

Hi all,

I’ve been playing with ArgoCD for the last two months. I’ve made some good progress along the way, but I currently feel stuck between a rock 🪨 and a hard place 🧱. To me it seems ArgoCD is leaning towards making Applicationsets the recommended way to work with ArgoCD above the app-of-apps pattern approach.

Looking at the current state of both approaches, I find it hard to decide which option to go for. They both seem to have their pro’s and cons. In case of apps-of-apps having an extra parent app to manage apps come across as hacky to be honest (Just seeing that app in the UI next to the other apps feels weird). ApplicationSets on the other hand are still a bit limited in their level of flexibility / featureset.

#9437 (which is included in the v2.6 release) seems to be an interesting alternative approach to sync-waves, which was not possible before whilst using appsets.

My main issue that I have with Applicationsets is that they are not easily templateable enough. #11567 might solve that, but it also adds extra complexity. In an ideal world 💅 I would like to use appsets to generate applications, but allow for extra overlays per app (Perhaps through Kustomize) to easily customise things like syncPolicy and ingoreDifferences .

As of now I could alternatively write regular application manifests and then using Kustomize and some scripting to bootstrap a cluster, but then I would have no support for sync-waves or progressive rollouts in case of appsets. Adding a possibility for application dependencies (FluxCD has a similar approach) would probably make me pick this approach.

So to sum up why I feel stuck and why this kind of feels like a trilemma. Either I go for:

1. App-of-apps which feels deprecated and hacky, but I get full application flexibility through helm templating + sync-waves
2. ApplicationSets which currently is hard / complex to customise on a per app basis, but allows for progressive rollouts.
3. Use regular application manifests with Kustomize and some scripting, but have no sync-wave support across apps.

Interested to get thoughts from others within the community. Are there other approaches I maybe haven't thought about?

[jujubetsz]

I'm stucked in the same place... I work in a company with 100+ clusters and everyday we setup a new one.. Today we bootstrap our "cluster tools apps" using an internal automation that is not ideal. I was going for App-of-Apps pattern but noticed the same thing as you, that this is probably going to be deprecated in favor of ApplicationSet but (again) like you, i think applicationSet is not ideal for now. In my case, i need to install some tooling (Logging collectors, metrics collectos, APM agents etc) and i need to be able to use sync waves and most important, be able to upgrade this tools if needed.

[purduemike]

I would like ArgoCD to KEEP app-of-apps and even expand it to app-of-app-of-apps. Ideally, I manually apply one app (to ArgoCD) that points to a dir in my github repo where I have many app-of-apps configs. This way I don't have to constantly manually create app-of-apps in ArgoCD. This also gives the benefit of using sync-waves. This seems to be the most simple and straightforward implementation.
My preferences:

1. I hate having to manually create anything. If necessary, I'll do it once, but that's it.
2. My repo should contain all of my app-of-apps and ArgoCD should use that as the source of truth and sync from it.
3. Sync-waves work on the application level, so let me easily create many app-of-apps.

[jeremyj563]

This approach makes sense to me and I understand where you're coming from with keeping things as straightforward / intuitive as possible.

I would very much like to hear feedback from anyone who's followed this pattern long term though. Any major drawbacks, or gotchas to consider before going this route?

[crenshaw-dev]

Quick note on the state of apps-of-apps: #11892 (comment)

[JanisOrlovs]

I use aswell app-of-app-apps approach. In my case we apply base app of app using terraform (as base being done with TF) and argo keeps running the applications it self

[crenshaw-dev]

To be super clear: app-of-apps is not deprecated. The idea of deploying Applications (which are just Kubernetes resources) from another Application is fundamental to how Argo CD works. It would be difficult to remove even if we wanted to.

As for the hackiness: yes, it does have limitations, bugs, and idiosyncrasies. And ApplicationSets (or something else) may be better for some use cases. But all tools have limitations, bugs, and idiosyncrasies. Many of them are documented in issues, but more are always appreciated!

[jeremyj563]

Thanks for giving that insider assurance. Could I bother you though to provide maybe some of the more common limitations, etc with app-of-apps vs appsets? And with that - I realize you can obviously create app-of-apps within apps generated by appsets 😅 but I'd really like to just stick with one pattern as much as possible to help make things consistent for my team.

One of the main use cases I'm trying to work out right now (I'm on 2.8 btw) is the ability to deploy multiple instances of a given "Helm release definition" (i.e. one for each environment). Ideally I could have an entire Helm release defined with many "default overrides", then I could reference that definition as a "base" and for each environment apply some additional overrides that would take precedence.

I know the terminology I'm using here sounds just like how Kustomize works but it would be so much nicer IMO if I could accomplish this all with ArgoCD native functionality only.

[gaeljw]

I use app-of-apps and ApplicationSet at the same time: app-of-apps for the very first bootstrapping, with actually two levels of app-of-apps, and then AppSet for self service by the teams mainly.

Both are useful and complementary IMHO.

[2mas]

I do the same with a bootstrapping Application in the 'root' repository that contains an Application per team that points to team's repo. The team repo then contains ApplicationSets that generates Applications by folder structure for each /service/cluster/env.

I solved/went around the issue with not being able to have different syncPolicies by having multiple ApplicationSets to split the Applications, basically like this (flipping the condition in matchExpressions)

  generators:
  - matrix:
      generators:
      - clusters: {}
      - git:
          repoURL: <repo>
          revision: HEAD
          files:
          - path: apps/**/{{name}}/**/values.yaml
          values:
            serviceName: "{{path[1]}}"
            env: "{{path[3]}}"
    selector:
      matchExpressions:
      - key: values.env
        operator: NotIn <-- In on the other one with autoSync
        values:
        - dev

[gaeljw]

@2mas be aware that in terms of responsibility and security, if anyone can create AppSet in the team repo, then they can create App in any Project and as a consequence potentially create "special" Kubernetes resources like RoleBinding.

That was our first approach but following an audit, we moved to a solution where the AppSet are owned by an "ops" repo that enforce the Project and people with access to the team repo can only create a minimal yaml (used by a git file generator AppSet) with few infos (like path, name...) but they cannot choose the Project.

[2mas]

Thanks for advice, I'll have a look at this approach also. It also might make it easier for the teams not having to worry about constructing the AppSets and coming up with different solutions in the end. As long as the AppSet is solid and fulfills the needs

[nastacio]

AppSets

Good for instantiating the same Application structure with different parameters stored in Git repositories

Example use case: Managing the same product across multiple cloud regions or across stages in a Continuous Delivery pipeline (dev/test/stage/prod)

App-of-Apps

Good for instantiating disparate Applications that make up a larger Applications.

Example use case: Deploying a product that a frontend application backed by multiple micro-services and data stores.

Net:
These use cases are complementary. I typically use AppSets at the top layer and App-Of-Apps as the second layer.
Top layer ensures all regions have the product.
Second layer ensures the product is deployed consistently within region.