Best practices for promotion between clusters

[musabmasood]

Hey GitOps!

I am struggling to understand how promotion between environments can be configured in a more GitOps fashion. Our use case is rather simple, we have 3 clusters (dev, staging, prod).

Application X is a simple helm chart which needs to be deployed in the following fashion:

1. auto-deploy to dev cluster (autosync)
2. trigger some smoke tests
3. auto-deploy to staging cluster
4. trigger smoke tests
5. manual approval
6. deploy to production
7. trigger smoke tests

Application source code and Dockerfile are in source.git repo. The helm chart with values-<cluster>.yaml are in deploys.git repo.

Side notes:

• Consider that autosync is turned on for the application in all clusters i.e. committing to master will trigger deploy
• We don't use branches per cluster, instead we use value files per cluster and watch master branch
• Not using app-of-apps or ApplicationSet at the moment
• ArgoCD is installed on 3 clusters for simplicity but we might switch to centralized one if the number of clusters grows

---

There are several strategies that I have thought of but I'm a little confused as to whats the best approach.

Strategy 1: PostSync job does remote git commit

1. CI from source.git makes a remote commit to deploys.git/application-x/values-dev.yaml with the new image tag
2. ArgoCD syncs the application on dev
3. PostSync job runs smoke tests
4. PostSync job remote commits on deploys.git/application-x/values-staging.yaml with the image tag
5. ArgoCD syncs the application on staging
6. PostSync job runs smoke tests
7. PostSync job creates a PR to update the image tag on deploys.git/application-x/values-prod.yaml which someone has to approve and merge manually
8. On merge, ArgoCD syncs the application on prod
9. PostSync job runs smoke tests

This looks neat but now I need to inject a post-sync remote committing job in all my application helm charts. I also need to give write access to that job and somehow manage git SSH keys in the cluster.

Strategy 2: PostSync job triggers committer workflow

This is the same as above but of doing the remote commit in the post-sync job, we trigger a workflow that does it for us. So step (4) above becomes:

4. PostSync job triggers workflow that will make a remote commit to application-x/values-staging.yaml

The workflow could be a jenkins pipeline, argocd workflow, some ad-hoc CI pipeline etc which takes parameters like:

serviceName: "application-x"
imageTag: "1.2.3"
env: "staging"
commitMessage: "auto-commit"

The job will basically update deploys.git/application-x/values-staging.yaml setting image.tag: 1.2.3 and then

git add application-x/values-staging.yaml
git commit -m ${commitMessage}
git push

Similarly another job could perform step (7) of creating a PR.

This approach is nice because I delegate the commit logic to an external engine. However now the devs need to keep an eye on it.

Strategy 3: Image updater pattern

This is a pattern introduced in Flux 1 (and I think its also now supported in Flux 2).

In this pattern, a controller watches image updates in a container registry (e.g. ECR) for updates, and (based on some policies) updates image.tag in git (writes to git) which in-turn triggers sync.

Lets call the controller "image updater"

In our case, I could think of it in this way:

1. CI pipeline pushes application-x:dev
2. Image updater detects the change and git commits the new SHA on application-x/values-dev.yaml
3. ArgoCD triggers sync on dev and runs some smoke tests
4. Once done, a post-sync job triggers a workflow that will tag application-x:dev as application-x:staging and push it
5. Image updater detects the change and git commits the new SHA on application-x/values-staging.yaml
6. ArgoCD triggers sync on staging and runs some smoke tests
7. Once done, a post-sync job triggers a workflow that will tag application-x:staging as application-x:prod and push it. In this step would could also just create a PR and to do so
8. Deploy to prod
9. Smoke tests

In this approach, tagging images essentially means promoting them.

State of things

I definitely want to learn how the community is attacking this problem. My concerns are:

If we're triggering workflows/pipelines after syncs, this means ArgoCD becomes a "step" in the CD process. Then a CD tool like Spinnaker makes more sense where argocd sync could be a "stage" during the whole deployment?

How do we provide a single workflow view to the developers who want to know at what stage their application is at the moment?

I have also looked at https://github.com/maruina/argocd-progressive-rollout-controller which looks promising but its not there yet.

[rumstead]

I am also very interested in how everyone handles this. Currently, we store the application CRs in a central git repo that ArgoCD watches. We are going to probably do the same when we move to Application Sets. Development teams control their deployment repos which contains all their Kubernetes manifests. Then developers create PRs (we have some tooling around this to make bulk updates and such easier) to the Application CR repo to point to a different git tag or commit sha of their deployment repo. The tooling will auto merge our preproduction environments and anything to production requires an approval from our CM team.

[jannfis]

Hey @musabmasood, there are two projects in the Argo CD ecosystem that might be of interest to you for accomplishing this use-case:

1. Argo CD Image Updater (https://github.com/argoproj-labs/argocd-image-updater) will take care about updating the container images used in your applications. The upcoming version v0.10.0 will be able to track mutable tags and update them using the image's SHA digest, so you will be able to track updates to a single tag (e.g. dev or staging)

2. Argo CD Notifications (https://github.com/argoproj-labs/argocd-notifications) will hook up on changes to your application's state and allows a variety of actions when your application transitions from e.g. OutOfSync -> Synced. You could configure a web hook that triggers a smoke test of that deployment, and if successful, promotes your image to the next stage by tagging it appropriately, so that the Argo CD Image Updater will pick it up.

I'm not entirely sure that it will cover all of your requirements, but it's at least an option to think about :)

[musabmasood]

Interesting. I was wondering what are your thoughts on #6114

[joebowbeer]

@musabmasood There are several steps in your plan that may need rolling back and you didn't explain how that will be accomplished. Have you looked at Argo Rollouts (analog of Flux's Flagger)? Can you let Rollouts do the syncs, using the smoke tests for validation?

I read that some teams will automatically create PRs to promote the new version to each subsequent stage. You could use a successful rollout to approve the subsequent PR. The production PR could require both a successful staging rollout and manual approval.

GitHub PRs can now be configured to merge automatically when all checks have passed.

I think Argo Notifications can report rollout status to GitHub.

[timothyclarke]

This is an extension to the question rather than an answer.

Can someone from the Argo project create a worked example of this with whatever parts of the Argo platform? Ideally also update the docs. My use case is is very similar to the original question except as there are some older apps involved a manual hold/approval is required prior to stage and prod deploys.

The entire reason I started looking at Argo was because multiple youtube video's implied that people are performing the actions described in the initial question of this thread.

[musabmasood]

Since ArgoCD considers git as the source of truth, we figured that every automation we want to do with it essentially boils down to git commit && git push. This applies to everything for instance:

• Promotion to the next environment
• Rolling back a failed deployment
• Automatically creating PRs for production deployment

Based on @jannfis suggestion, we are using ArgoCD Notifications to take actions based on the deployment status. For instance:

• When deployment on staging succeeds, we auto-commit to the next environment values file to trigger a deployment
• When all environments succeed, we automatically create a PR to for production values file

This is the best we have been able to accomplish so far, but there are lot of edge cases:

• Too many git commits for image tags updates or non-prod environments
• git conflicts

[OferPRTZ]

Since ArgoCD considers git as the source of truth, we figured that every automation we want to do with it essentially boils down to git commit && git push. This applies to everything for instance:

• Promotion to the next environment
• Rolling back a failed deployment
• Automatically creating PRs for production deployment

Based on @jannfis suggestion, we are using ArgoCD Notifications to take actions based on the deployment status. For instance:

• When deployment on staging succeeds, we auto-commit to the next environment values file to trigger a deployment
• When all environments succeed, we automatically create a PR to for production values file

This is the best we have been able to accomplish so far, but there are lot of edge cases:

• Too many git commits for image tags updates or non-prod environments
• git conflicts

Hi Musabmasood,
thanks for the update, we are also took this approach but kind of struggling with it.
are you using argo-workflow to auto-commit the next env? or can you please share your flow? will be much appreciated!

thanks

[musabmasood]

Since ArgoCD considers git as the source of truth, we figured that every automation we want to do with it essentially boils down to git commit && git push. This applies to everything for instance:

• Promotion to the next environment
• Rolling back a failed deployment
• Automatically creating PRs for production deployment

Based on @jannfis suggestion, we are using ArgoCD Notifications to take actions based on the deployment status. For instance:

• When deployment on staging succeeds, we auto-commit to the next environment values file to trigger a deployment
• When all environments succeed, we automatically create a PR to for production values file

This is the best we have been able to accomplish so far, but there are lot of edge cases:

• Too many git commits for image tags updates or non-prod environments
• git conflicts

Hi Musabmasood, thanks for the update, we are also took this approach but kind of struggling with it. are you using argo-workflow to auto-commit the next env? or can you please share your flow? will be much appreciated!

thanks

Currently we are using ArgoCD Notifications webhook for successful deploys to create the PR for next environment.

[OferPRTZ]

Since ArgoCD considers git as the source of truth, we figured that every automation we want to do with it essentially boils down to git commit && git push. This applies to everything for instance:

• Promotion to the next environment
• Rolling back a failed deployment
• Automatically creating PRs for production deployment

Based on @jannfis suggestion, we are using ArgoCD Notifications to take actions based on the deployment status. For instance:

• When deployment on staging succeeds, we auto-commit to the next environment values file to trigger a deployment
• When all environments succeed, we automatically create a PR to for production values file

This is the best we have been able to accomplish so far, but there are lot of edge cases:

• Too many git commits for image tags updates or non-prod environments
• git conflicts

Hi Musabmasood, thanks for the update, we are also took this approach but kind of struggling with it. are you using argo-workflow to auto-commit the next env? or can you please share your flow? will be much appreciated!
thanks

Currently we are using ArgoCD Notifications webhook for successful deploys to create the PR for next environment.

Thanks for the quick response!
got it! can you please share some config or how to do it? i cant actually find any docs or examples how to to make argo notification trigger a workflow.

much appreciated

[NickLarsenNZ]

Thanks for the quick response!
got it! can you please share some config or how to do it? i cant actually find any docs or examples how to to make argo notification trigger a workflow.

much appreciated

@musabmasood, I'd also be interested in seeing an example config for this.

[musabmasood]

So this is our setup right now since a few of you asked about it. Our repo looks something like this:

applicaitons/
   myapp/
     values-staging-images.yaml
     values-staging.yaml
     values-qa-images.yaml
     values-qa.yaml
     values-produciton.yaml

• We currently use github
• Our home grown applications are deployed as helm charts and value files are named as values-<cluster>.yaml. The value files live within the helm charts.
• We don't use branch per cluster model, everything is in the main branch
• values-<cluster>.yaml is maintained by humans, mainly service owners and SREs
• No one is allowed to touch the values-<cluster>-images.yaml files, its always updated by CI bots
• values-<cluster>-images.yaml has only one key: imageTag which is updated by CI tools
• We enable autosync on all deployed applications in all clusters (including prod)
• The Application.yaml for myapp takes two value files in this order:

valueFiles:
  - values-<cluster>-images.yaml
  - values-<cluster>.yaml

This is how our deployments and promotions work:

1. The CI tool, after building/pushing image to registry, creates a PR to update imageTag: <newvalue> in values-staging-images.yaml. The PR and branch for it are auto-generated. The workflow for this task is created in GitHub actions, lets just call it update-image-tag. It takes a cluster name and a new value for imageTag.
2. The PR is then auto-approved and merged by a "cd-bot" user who is the code-owner on all *-images.yaml files in the repo
3. After the merge, since autosync is enabled, staging gets deployed with imageTag: <newvalue>
4. On a successful sync, we trigger integration tests against the deployment. This is a simple post-sync hook.
5. When the integration tests are also successful, ArgoCD will consider the sync as "Successful". We then use ArgoCD notifications trigger.on-deployed to trigger the update-image-tag GHA again, but this time with qa cluster and provide the same <newvalue>. Steps 1 to 4 are then repeated for qa cluster.
6. For production, its a manual PR process so we don't have an images.yaml file which is bot updated.

Pros:

• Git is the real source of truth, every change and deployment comes through git and no "deployment pipelines"
• WYSIWYG - Whatever is on git is deployed since autosync is always on

Cons:

• No autorollbacks at the moment
• We have to be careful with the concurrency of update-image-tag GHA since it can sometimes lead to git conflicts etc.
• No single end-to-end pipeline for devs to view (this is a big one, since the whole CD process is more event based and not linear).

Would love to hear your thoughts and what you are doing about promotions.

[kinluek]

I like this approach! What do you use for the git bot that creates and approves PRs etc? Have you written your own scripts using github api? Or is there something out there that can do this without much coding?

[fabioaraujopt]

What if you make changes to the helm chart itself, for example creating a new custom CRD. How do you ensure you have a version of the helm chart in DEV and other in QA and how do you promote the helm chart version itself?

[WFA-hhsieh]

What if you make changes to the helm chart itself, for example creating a new custom CRD. How do you ensure you have a version of the helm chart in DEV and other in QA and how do you promote the helm chart version itself?

I also struggling on the topic.
Anyone has the idea to use different App Helm Chart version in each environments and promote?

[todaywasawesome]

Not directly answering the question but related is the Argo CD Autopilot project that specifically provides patterns for promotion across environments.

[Hollerweger]

We are using following structure (no GitOps but Infrastructure as Code).

Some of our requirements (only the ones not clarified in GitOps):

• It should be possible to identify a Platform version for all environments on a single git commit
  • For better auditing and tracing
• Promotions between environments should be easy and can be automated
  • Merge conflicts need to be avoided
• Avoid that any change intended for a non production environment can influence production
  • Changes on Git need to be first validated on non production environments

Which leads to following Question:

• What are the environmental differences for one Platform version what are the similarities?
  • All service versions should be the same for a specific platform version
  • Environment configuration is different per environment
    • Typical service/application properties/settings
    • Egress/Firewall, ...
    • Services are maybe enabled or disabled on specific environments

Therefor we came to following approach:

• Environmental config values are separated per Environment folder
• Service/Application versions (docker-tags) are same and stored in a single file for all environments.
• Deployment is triggered from environmental branches
  • One branch for each environment
• Only allow fast forward merges to environmental branches

The benefits of such a solutions are:

• Platform Release for all environments can be linked/tagged to a git commit
  • Git tag can be a platform version for all environments
  • Better tracing and auditing
• Promotions from one to another environment can be done via a PR/merge
• Testing changes on infrastructure only affects intended environment (Branch)

Downsides:

• Changes on environmental branch (eg.: production/qa) directly should be avoided or always be merged back to base branch (e.g: develop environment) to avoid any future merge issues
  • Merge issues are mitigated by only allowing fast forward merges to environmental branches

Questions;

• What do you think about such a structure?
• Would it also be a good fit for GitOps?

[willemm]

You said:

Counter question: Do you know what fast-forward merges are?
In other words: IF you enforce fast-forward merging, then promotion is always a simple git merge!

But now you say

Also there is the thing about fast-forward merges, which trivially cannot work if you have environment-specific differences between branches.

So I am confused. Can you do ff merges or not?

"if you have environment specific differences between branches" means "if you're using the anti-pattern you described in your article". So not the model I'm proposing. In my model, you can and should.

But in the your scenarios
you say:

If you really want to do this, however, this is possible with a git cherry-pick, by searching for the commit in which the build pipeline committed the newest version.

So I am confused. Are cherry picks required or not for some scenarios?

I only figured out yesterday how you can have out-of-order promotions if you really need to, and still get back to the 'clean' state where you can do fast-forward merges with a quite simple approach, even. It's still a bit complicated, but that's good because out-of-order promotions should be discouraged. I still believe they should be outright banned but if someone wants to do it, it's no longer impossible in my model.

So I make my change in the QA folder/file environment in the QA branch first? Is this how it works? And then if I merge to integration, my change is still in the QA folder/file. Correct? How does this work? Since you now clarified that you mean both folders AND branches can you update your scenarios to explain which folder/file you also touch with each change (and how each change moves both between folders AND between branches)? The changes move between branches with merge operations. How do they move within files/folders?

They don't.

You make your change in the base folder of the main branch, which then gets promoted to the qa branch. I'll update the scenario description tomorrow. I thought this was clear from the general description, and also from the fact that the settings.yml and version.yml have disappeared from the envs folders.

Edit: Actually I found it

So, go to the main branch, copy the environment-specific replica setting from staging-asia to qa, and then running the default promotion flow to pass it to qa

So it seems you do need copy operations for some scenarios after all.

This is a specific description of the scenario "copy environment-specific settings (e.g. replicas) between environments", which is far-fetched at best. I see no real-world scenario where you'd have the requirement to copy such specific settings between environments, rather than just updating them per environment.

By using both folders and branches you instantly have extra complexity. For 20 environments you will have a matrix of 20 branches plus 20 folders/files. Not only that, but you also need to be careful to match them. If you need a change to land in QA you need to make the change to the correct folder AND branch at the same time.

Almost all of the changes go in the base folder. The complexity of the envs folder is much reduced. It only contains those settings that are (permanently) unique to each environment. If you need a change to land in QA, you make the change to the base folder, just like for all the other environments. So no "extra complexity" there.

1. Heavy file duplication. For example the QA folder/file in the "staging" branch is not important most of the time. It doesn't actually show what is in the QA environment. Correct?

Partly correct, but there is NO "heavy file duplication". In fact, the file duplication is greatly reduced, because only environment-unique settings exist in different folders. All branches are actually the same (except some are ahead or behind in time).

2. All changes must be promoted in the order they come. It is impossible to promote features out of order. You consider this a good practice. I consider it a limitation (especially on product companies where business might change order of features literally in the last minute). Let's agree to disagree there.

This is only possible for out-of-order promotions of configuration changes anyway. What I would describe as "features" would usually live in the application source code. Also, it's possible (but I would not recommend it) to do out-of-order promotions using the cherry-pick method mentioned earlier. I thought it was not possible but I found a way to "undo" the "damage" caused by cherry-pick merges (i.e. by git reset).

3. It is impossible to promote features until a certain env. They will go all the way to production starting from the first environment

It is possible.

4. It is impossible to skip environments during promotion

It is possible.

5. It is very hard to do hotfixes in production (this is also in your scenarios)

I would not describe it as "very" hard. Also, I'm not sure how you would handle hotfixes in your model so I can't compare.
Even then, it is quite simple to automate this procedure.

6. It is very hard to do backportings or different order of promotions

In any case, it's not at all hard to do backportings. The only scenario in which backportings would be necessary is if you skipped an environment earlier, and then it's a simple merge between the two environments. Otherwise, the need to backport would never arise because you're not allowed to make changes directly in "higher" environments.

(I think you said (at point 2) that different order of promotions was impossible. See my answer there.)

7. Some operations such as rollbacks require multiple commits in multiple branches (this is also in your scenarios)

No, they require git resets, in exactly as many environments as need to be rolled back. Which is really really easy, it's basically like an 'undo' button. Whereas in your model, you can only roll back if that was the latest change (otherwise you need to revert commits which is a lot more complicated and prone to errors and merge conflicts).

8. Most promotions can be done with ff merges. Some will require cherry-picks

Only if you want out-of-order promotions, and then only those promotions that are actually out-of-order. Also, in your model those same promotions would require manually picking which files need to be copied instead of relying on the auto-promote script.

9. Most promotions can be done with ff merges. Some will require copy operations

No promotions require copy operations. The "copy replica settings example" is not real-world.

So for this complexity and limitations the only thing that you gain is to better enforce security on a branch level. Correct?

These are some of the benefits of the branches+env folders model:

1. All promotions between environments use exactly the same commands. There is no need to 'teach' your promotion scripts the specific folder structure of your applications.
2. File duplication is greatly reduced. Only settings that are unique to an environment (or variant) live in a separate folder.
3. Configuration complexity is somewhat reduced, because you need to reference less files.
4. Overall complexity of the gitops repository is reduced quite a bit. Even more so if you use helm (which has other benefits I won't go into now).
5. If you wish to know which changes are on a specific environment, you can see that directly in the git history of that branch.
6. Same goes for promotions that are done with pull requests; you get a comprehensive list of all changes present in this promotion. (You can easily get the same with any other mechanism that involves a human).
7. Environments can be secured almost out-of-the-box, simply by leveraging existing code review tooling.

Additional benefit which only applies when disallowing out-of-order promotions and also disallowing skipping environments in the promotion flow:

8. Any change that makes it to production has been integrated and tested in conjunction with all other changes that are there. No risk of two changes being deployed which have never been tested together.

[kostis-codefresh]

I will answer your points one by one but I guess the major disagreement is that you consider "out-of-order" promotions as an anti-pattern. Usually in product based companies there are multiple features in flight and business might want to release feature C right now. Instead of answering them "no we have to release A and B first", it is better to have an option to do it correctly. Maybe you don't have this problem in your organization but other organizations need this.

You make your change in the base folder of the main branch, which then gets promoted to the qa branch. I'll update the scenario description tomorrow.

Actually it is not that simple. You have to make the choice in advance of where to put the change. It you put it in base (or respective Helm file) it is assumed that will be promoted in all environments. If you want it to stay in an env, you need to put it in the overlay. With the single branch model you can put it anywhere you want and then decide later during the promotion time if you are going to take it or not in the next environment.

which is far-fetched at best. I see no real-world scenario where you'd have the requirement to copy such specific settings between environments

This is a realistic scenario in some organizations.

Almost all of the changes go in the base folder.

Exactly, see my previous point where you have to know in advance where to put the changes. There is no such limitation with a single branch.

It is impossible to promote features until a certain env.

I do a change in base and then it goes all the way up to integration. But then they tell me that the feature needs to stay there. So now the next feature comes in and once the merge from integration takes place it will also move that feature as well (which I don't want to). So I have to manually intervene and move that feature I don't want back to an overlay (using .....copy operations)

It is impossible to skip environments during promotion

Same point as before. If a change is in base and I want it to skip an environment after the initial commit, I need to convert it manually to an overlay so that it doesn't get picked with the next feature.

I would not describe it as "very" hard.

Your example README has many bullet points. Do you have a script that does it? Also it seems that every time I add a new environment in your model I would have to check all my scripts that do something with environments and find any loop instructions and update them. With a single branch the hotfix is trivial. You just go to production folder and commit. Nothing else to do. It doesn't get any simpler than that.

otherwise you need to revert commits which is a lot more complicated and prone to errors and merge conflicts

I find "git revert" less dangerous than "git reset".

Regarding your list "of benefits of the branches+env folders model:"

1. "Common commands". I some cases you will have git reset and in some cherry-pick and in some other you will need cp operations to move stuff from base to an overlay. On the other hand with a single branch is always cp
2. Same for single branch. Normally only overlays have extra stuff
3. I believe that 1 branch and 20 folders is less complex than 20 branches with 20 folders.
4. Same as above
5. Same for single branch. You go the folder of the environment and see what is in there. And if you look at git history of that folder you see what was changed.
6. Not sure what you mean by pull requests here. You can have pull requests even with a single branch
7. "Environments can be secured almost out-of-the-box". Actually you now have the danger of me committing in an unrelated branch in an unrelated folder. So you still need some protection in the folder level as well.

[kostis-codefresh]

@joebowbeer

1.5 Folders/files per environment (single branch) AND tags

That is a very interesting idea as well. I would certainly use tags in some cases to mark/denote major releases events. But maybe only for historical values. I am not sure how one should use tags as part of the model itself....

[willemm]

I will answer your points one by one but I guess the major disagreement is that you consider "out-of-order" promotions as an anti-pattern. Usually in product based companies there are multiple features in flight and business might want to release feature C right now. Instead of answering them "no we have to release A and B first", it is better to have an option to do it correctly. Maybe you don't have this problem in your organization but other organizations need this.

We're not talking about features though, are we? We're only talking about gitops settings. Features come from the source code, so all features come in the form of docker image versions. It's an anti-pattern and can go wrong.

Also, I've had enough managers trying to pressure the devs to completely skip the testing phase because "we ran out of time and this needs to be in production asap".

You make your change in the base folder of the main branch, which then gets promoted to the qa branch. I'll update the scenario description tomorrow.

Actually it is not that simple. You have to make the choice in advance of where to put the change. It you put it in base (or respective Helm file) it is assumed that will be promoted in all environments. If you want it to stay in an env, you need to put it in the overlay. With the single branch model you can put it anywhere you want and then decide later during the promotion time if you are going to take it or not in the next environment.

With the single branch model, you have a settings.yml file in the overlay. That's the file you're copying from environment to environment. If you suddenly decide that a certain setting needs to not be promoted, you have to separate that setting out from that file and put it into a separate file.

So this is at least as, if not more, difficult in your model. And it greatly increases the number of files and the complexity of the setup.

And in any case, this is a very rare occasion. 99% of the time, features will get promoted through all environments.

Not to mention it's a bad idea to have configuration drift between environments. Environment-unique settings should be kept to a minimum, so that tests remain valid. Otherwise you get "it works on my box" scenario's.

which is far-fetched at best. I see no real-world scenario where you'd have the requirement to copy such specific settings between environments

Dismissing my scenarios as "far-fetched" implies that you know how all organizations work and what scenarios they need. This cannot possibly be true.

I have no idea how you're making that inferrence.

Almost all of the changes go in the base folder.

Exactly, see my previous point where you have to know in advance where to put the changes. There is no such limitation with a single branch.

Yes there is, as explained previously.

It is impossible to promote features until a certain env.

I do a change in base and then it goes all the way up to integration. But then they tell me that the feature needs to stay there. So now the next feature comes in and once the merge from integration takes place it will also move that feature as well (which I don't want to). So I have to manually intervene and move that feature I don't want back to an overlay (using .....copy operations)

Businesses do not want features to permanently live on the testing environment but never make it to production. See my previous point about invalidating tests and "works on my box".

It is impossible to skip environments during promotion

Same point as before. If a change is in base and I want it to skip an environment after the initial commit, I need to convert it manually to an overlay so that it doesn't get picked with the next feature.

Why would you want that? This promotes configuration drift between environments!

Also, please explain how this works in your model once you introduce automated promotion flows. "So now the next feature comes in and once the promotion flow from integration takes place it will also move that feature as well".

I would not describe it as "very" hard.

Just look at the size of the paragraph in your own README file and how many bullet points it has. Do you have a script that does it? Also it seems that everytime I add a new environment in your model I would have to check all my scripts that do something with environments and file any loop instructions and update them. With a single branch the hotfix is trivial. You just go to production folder and commit. Nothing else to do. It doesn't get any simpler than that.

"I don't always test my code, but when I do, I do it in production"

Also, if you do that, your hotfix will be lost the next time you do a regular promotion.

Regarding your list "of benefits of the branches+env folders model:"

1. "Common commands". You already said it yourself that in some cases you will have git reset and in some cherry-pick and in some other you will need cp operations to move stuff from base to an overlay. On the other hand with a single branch is always cp

In my case, the only command I need is 'git'.

If you think there's something wrong with that answer, think about why, and why that same reason applies to 'is always cp'.

2. Same for single branch. Normally only overlays have extra stuff

No, in your case the overlays have all the settings that are promoted between environments. In your example, the settings.yml and the version.yml. Duplicated. In each folder.

3. If you believe that 20 branches with 20 folders is less complexity than 1 branch and 20 folders, then frankly I don't know what to answer there.

Do you believe that 20 branches with 60 files is more complexity than 1 branch with 100 files ?

4. Same as above

Please explain.

5. Same for single branch. You go the folder of the environment and see what is in there. And if you look at git history of that folder you see what was changed.

In single branch, if you look at the history log of an environment folder, all you will see is a long list of commits saying "promoted from qa to testing". In the multi-branch model, you get the actual commit logs of the changes that were done.

6. Not sure what you mean by pull requests here. You can have pull requests even with a single branch

But you don't get to see which git commit messages are part of this change, so you don't know which features (e.g. jira items) are actually part of this PR.

7. "Environments can be secured almost out-of-the-box". Actually you now have the danger of me committing in an unrelated branch in an unrelated folder. So you still need some protection in the folder level as well.

Please explain how any change "in an unrelated branch in an unrelated folder" can make it to the protected environment without passing through the protection.

Also some more points:

With branch per env, you can simply promote region-specific changes through qa-staging-prod.

The thing that needs an enormously complex step of operations comes for free, out of the box, in the branch-per-env model.

So, to recap. The drawbacks of one-branch:

• You have the freedom to introduce configuration drift between environments (which is bad).
• If you want, you can keep certain features out of production eternally (which is bad).
• As soon as you start automating, most "benefits" go out the window.
• Hotfixes need to be manually applied to all the environments, instead of going through the normal flow.
• You can't just pick and choose which features to promote, because you are limited by which settings are in the same overlay file.
• Very hard to promote region-specific variant settings.
• It's remarkably easy to shoot yourself in the foot.

[kostis-codefresh]

For anyone that comes across this discussion I will also link the guidelines from Flux (the other major GitOps project) and their example repo at https://github.com/fluxcd/flux2-kustomize-helm-example.

[joebowbeer]

@Hollerweger the reasons against using (GitOps) branches presented in this article make sense to me

https://codefresh.io/about-gitops/branches-gitops-environments/

[nastacio]

I agree with no branches per environment, with env-specific files, but versioning the repository is essential to avoid a PR-fest to update the env files (one PR per env.)

[Skalador]

Very nice article, this made sense to me.

[Hollerweger]

@joebowbeer Without using environment branches I see then all the issues that are presented in this blog about GitOps: https://codefresh.io/about-gitops/pains-gitops-1-0/

Especially these points:

• GitOps doesn’t address promotion of releases between environments
• There is no standard practice for GitOps rollbacks
• Observability for GitOps (and Git) is immature
• Auditing is problematic despite having all information in Git
• Running GitOps at scale is difficult
• GitOps and Helm do not always work well together

My suggestion from above is using both environmental branches together with folders per environment.
Regarding the Blog post you shared: I tried to especially handle the usual issues with environmental branches with the example above.

[joebowbeer]

@Hollerweger both blog posts are by same author. The one you cite is over a year older. Prehistoric, that is, in terms of GitOps. I tried to use branches back then too, based on some early demos, but abandoned it. I guess we're both waiting for that next blog post...

Check out how ArgoCD rolls out within Intuit: https://youtu.be/ESQLqjbM8h0

[Hollerweger]

The same issues mentioned there are still applicable today - even if a one year old post is prehistoric for GitOps. At least I haven't seen how they are solved in a good way.

[kostis-codefresh]

They are still true today indeed. We (Codefresh) have been trying to solve them with this open-source project https://github.com/argoproj-labs/argocd-autopilot But we are a long way from solving everything mentioned in my article.

[musabmasood]

We decided not to go with the "branch per cluster model" and instead "values-file-per-cluster" for now (which will be replaced with ApplicationSets soon). Nothing against the branch-per-cluster strategy but:

• Doesn't work very well when you have many clusters
• Merge conflicts / cherry picking when merging
• Automated promotions are hard when automerging from branch to branch (conflicts)
• Automating branch restrictions

[romuduck]

I'm using applicationSet a lot... It does not replace my env files. How do you plan to use it?

[musabmasood]

We haven't yet started using it. It doesn't solve the promotion problem. However it does help in bootstrapping multiple clusters with the same "system services" which I think is really useful.

[Hollerweger]

@musabmasood and @joebowbeer: I see the same drawbacks when only using branch per envionment model. In my example i combined "branch per environment" & "folder per enviroment" to solve the main issues from both models. I don't see in your reponses that you responded to this idea.

Haven't heared about ApplicationSets - need to understand what it tries to solve.

I tried to update the example above and explain it in more detail: #5667 (comment)

[kostis-codefresh]

I guess we're both waiting for that next blog post...

Hello. @joebowbeer

I am the author of the 2 Codefresh articles. Thanks for mentioning them. They have opened indeed a great discussion on how to structure repos/how to promote an app.

Here is another recent view on the same subject that also briefly mentions the other approaches. Hopefully you can get some ideas until I finish writing my own proposal for environment-per-folder

https://en.sokube.ch/post/promoting-changes-and-releases-with-gitops

[romuduck]

Btw @kostis-codefresh, the doc you point up here is also suggesting to add branches on top of a folder/file set-up when env files are no longer sufficient (which is always in real life with super critical applications)

[kostis-codefresh]

I never said I agree 100% with everything written on that article. I linked it because I think it is a good overview of all the choices people might use when it comes to organising gitops environments.

Anyway, my own detailed article is almost ready. I will post here when it is live

[romuduck]

thanks @kostis-codefresh , cant wait :-)

[joebowbeer]

This kustomize guide from OpenAnalytics is worth checking out:

https://www.openanalytics.eu/blog/2021/02/23/kustomize-best-practices/

It makes a case for maintaining bases in a separate repo, in larger projects, and even maintaining a separate repo per environment.

Quoting from the Structuring Kustomize Repositories section:

Kustomizations should be divided into bases and environment-specific (live) overlays.

One way to implement this division is as top-level directories bases/ and overlays/. This is the standard approach typically found in kustomize examples and works well for smaller projects.

Another approach is to use separate repositories: a base repository and a live repository. We use the term ‘live’ here instead of ‘overlay’ since a kustomization can be both a base and an overlay. ‘live’ communicates the desired intent better: an environment-specific overlay that describes the final form of the infrastructure and should correspond one-to-one to what is deployed on the cluster.

Using repositories instead of folders has several advantages:

Bases can be re-used across projects. This helps to keep Kustomize DRY (Don’t Repeat Yourself).
The live overlays can be locked to a particular version tag of the base repository. This is especially useful to gradually promote improved bases across environments since each environment overlay can be locked to a different version. We recommend using a standard versioning approach like SemVer and independently versioning the bases by including them as a scope prefix in the tag: mybase-v1.2.3.

For some projects, it is useful to create separate repositories per environment. A primary reason for this is to allow for separate access rights per environment.

[kostis-codefresh]

@musabmasood , @joebowbeer , @Hollerweger , @romuduck

Here is my suggested structure https://github.com/kostis-codefresh/gitops-environment-promotion

Associated article with more details https://codefresh.io/about-gitops/how-to-model-your-gitops-environments-and-promote-releases-between-them/

[kostis-codefresh]

I don't have a public example right now. But it is standard practice in many companies I know.

See the codecov report as a similar example #9049 (comment)

Imagine the same thing but with a diff from manifests instead of source code. Same thing for kubeval, datree, kics, etc. or whatever tool you need for checking manifests

[rumstead]

The way I would handle

If I want I also attach on the PR of the full diff of the prerendered manifests for extra validation

Is by commiting the rendered manifests to the same repo. Only 1 configuration/manifest repo

[olljanat]

So would I but sure it can be done on way that those manifests would be stored example to "for-pull-request-review-only", etc folder and never used for deployment (so we don't break GitOps principles). However your CI still need to verify that those those who created PR remembered update prerendered manifests which is also a bit ugly solution. Better way probably would be move those prerendered manifests history to another repo (which no one updates manually), let CI open pull request against of it and use something like https://www.dpulls.com to make sure that that PR gets merged first.

What comes to handling promotions I think that @nastacio proposal to use environment specific tags on "targetRevision" and update those it Git is most probably way to handle them. However there you then need very carefully check that who have rights to update tags in main branch.

[musabmasood]

Yes. The article assumed auto-sync. This is the preferred way as far as GitOps is concerned.

• Rollbacks. Up to you. Do git revert/reset or rollback forward by doing another point release. I don't see an issue here
• What is the use case for that? You mean pause CD while you still push source code? why would you want to do this? Maybe use ArgoCD sync windows? I would love to know more about the use case here
• If conflicts you mean git complaining that commits touch the same lines, this should never happen. You don't want different releases to go in the same environments (and certainly not in production). If by conflict you mean that git complains that you need to pull again before commiting, then just retry in a loop or serialize all deployments on the CI level
• Not sure what is the problem with history. You can look at Git history on a single file/folder level so you get only those commits that affected it (and therefore only those promotions that touched it
• I talked about this in the end of the article. You need to use an external system that commits on your behalf
• Same answer as above. I don't expect humans to actually copy files around for each promotion. It should be an automated system and humans should only intervene in emergency hot fixes.

Please link your article here when you write it :-)

So this is interesting (I believe you also wrote that article?):

To create a CD pipeline in Codefresh that is responsible for GitOps deployments you 
must first disable the auto-sync behavior of ArgoCD

I would argue that here you are going against the GitOps philosophy by not using the ArgoCD GitOps controller to monitor changes but using a standard git trigger to run argocd app sync. I do agree that this makes the end to end pipeline easy, but I find thats cheating and going back to the old imperative way of deployments. Once you disable auto-sync you are basically out of GitOps in my opinion (also resonated by another CodeFresh teammate).

[kostis-codefresh]

Hey @musabmasood

I would argue that here you are going against the GitOps philosophy by not using the ArgoCD GitOps controller to monitor changes..

I agree 100%

The link you posted is not an article. It is documentation from the Codefresh classic product that describes end-to-end pipelines. Some people/organizations prefer that approach (with all the advantages/limitations it has).

If you look at the part labeled "Performing Automatic Git Commits" in the same page it has the other scenario where auto-sync is enabled instead and the pipeline only commits to Git.

[nastacio]

We have been versioning our repo automatically once tests pass in dev, then adjusting the "targetRevision" field in the respective Application(s) in the Argo CD instance managing the next stage of the pipeline.

Prod points at a "stable" tag, which always references the commit of the last revision validated in "stage".

All of the git versioning and tagging machinery is done inside GitHub actions.

I wrote an article with the basic principles: https://dnastacio.medium.com/gitops-repositories-the-right-way-part-1-mapping-strategies-6409dff758b5 (see section "Lesson 2: Versioning with branches and tags")

[musabmasood]

Very interesting ideas.

We have a mono-repo with all applications, value files (separate per env) living in it. We tried the git tag method with this convention: <service>-<environment> . We kept moving the tag whenever there was a new commit and ArgoCD would detect/resync.

However the problem with this approach was that the history of all changes for a particular application became difficult to understand. The HEAD of the deployments repo no longer reflects the desired state of the world but you need to filter to the right tag for that. Further more, moving the tags becomes trickier in case of rollbacks (also commits) etc.

At the moment we're pointing everything to HEAD but we stopped using autosync. I feel like without autosync we're just doing CIOps and not GitOps but thats the only real way to get an end to end pipeline (which devs love) from commit to deployment and promotion.

[jemag]

@musabmasood why not keep autosync and still go for the end to end pipeline? We just started using this hybrid approach to try and get benefits from both. Basically what we are going for in the pipeline is:

• Commit and push changes to dev manifests
• Run argocd app sync --async (ignore error if sync is already running)
• Run argocd app wait --sync and wait for it to be synchronized
• Verify sync status, health status and sync revision with argocd app get
• If everything is proper, execute smoke tests
• If smoke tests pass, repeat process for test and eventually prod

You could also gate with approvers using something like GitHub's environments or create PRs instead of having the pipeline modifying the code directly.

[musabmasood]

This is exactly the approach we finally went with!

We went with the hybrid approach with the tradeoff that image changes are not stored in git anymore (infact we have a CI check to prohibit adding them there). Although its not 100% gitops, its still good enough.

The problem with storing images in git:

• Way too many git commits and exponentially increase based on number of services x number of environments x frequency of deployments
• Rate limiting issues with the git provider (e.g. github)
• Do you push directly to master or automate a PR which gets merged? github is way too complicated to set this up without giving up security.
• Auto-resolving conflicts etc
• Thousands of commits everyday on your repo, history of non-images changes is hard to comprehend
• Keeping feature branches up-to-date with master becomes nearly impossible

Therefore we went with a hybrid approach:

• Autosync is still on for applications
• However, image tags are direct overrides on the application using argocd app set
• Deployments go through regular "pipelines" which we have full control over for example notifications, testing, rollbacks, promotions etc
• We have validations (e.g does the image tag actually exist? If deploy was successful, is the live image tag the same as what we desired?)
• We emit events to datadog when deploys succeed/failed and have a dedicated dashboard to search through deploy history

The advantage of this approach is:

• Non-image changes are immediately updated as autosync is on (e.g. updating a port, replicCount, helm chart change etc)
• Deploys go through regular pipelines which are easy to understand for devs
• Devs can trigger a deploy through the UI any time for any environment

So far, this is the best approach we have figured out.

[romuduck]

I've been using same approach as @jemag. However, an argocd app sync/wait was not enough as many apps may point to same git reference. So we build a kind of logic to list all apps listening to a given repo/ref and poll status...
I think some work on operators like #9437 would help removing pipeline from the picture.
Also I think we need something like the analysis step available with argo-rollout, but to be used always, as soon as app has been resynced

[fulcrum29]

Hey @musabmasood
I really like your approach, We are currently commiting new image tag for every deployment and I've noticed the issues you mentioned.
One question I have - How do you emit events to datadog? I would like to have grafana dashboard with deploy history to substitue for git repo history, ArgoCD notifications seems to be not sufficient for this. Have you created some custom tooling for that?

[ddeath]

I would like to point out that nowdays there is https://github.com/akuity/kargo which is taking care of promotion aspect.

I think it is kind of doing what all suggestions / examples here are doing:

• checking container registry for new tag
• committing new image tag to your argocd config file which triggers deployment
• runs verifications
• if verifications are successful and all conditions are met it will commit that tag to higher env config file

Basically it is doing what accepted answer suggest, you just define your cp commands in Kargo manifests. The benefit here is that you have UI slapped on top of it where you can easily observe what is deployed and what could be deployed (older artifacts) and much more...

At the time of writing this comment they just recently released first stable 1.0.0 version so expect:

• documentation not so great
• bug here and there

Maybe this will be helpful for someone.