Ephemeral/preview environments

[scottrigby]

In today's virtual GitOpsCon EU Day 1 Q&A, this question came up:

question by @n-029894
Any recommendations for using gitops for ephemeral/preview environments and specifically when testing changes that span multiple microservices?

There are solutions out there from multiple vendors and Open Source projects, some of which are done according to GitOps Principles.

reply by @monadic:

I have seen it working well with flux and I also see people using it in "argo based products"

Here is a slack thread with more replies on this topic from the CNCF Slack workspace #opengitops channel.

Let's continue the discussion here.

IMO this could be a good joint topic for multiple folks to collaborate on, with a goal of creating some content about this that could live in the OpenGitOps documents repo. Anyone interested in collaborating on that? If so, we could propose a content team focused on this topic. But let's start with this discussion here and see how much interest there is.

[scottrigby]

AKA "Pull Request preview environments" with GitOps. But left this topic more general in case others are interested in GitOps-based ephemeral/preview that may not necessarily be tied to PRs.

However I believe PR preview is the most common use case. Teams I've worked with have been doing that for at least 10 years, just not using GitOps. Folks who have done this in various ways with GitOps are encouraged to share their experiences in this discussion!

[n-029894]

The main thought I have been having is that ephemeral environments are pretty easy to do when it's just testing changes to a single app/microservice. It quickly gets complicated when you have to consider changes that span multiple microservices. There needs to be a streamlined way to specify version numbers or specific image tags and a way to group them in whatever PR ultimately gets opened to create that environment.

I have a number of ideas but they're all kind of disconnected pieces of the puzzle though. Not sure how to put them together. If anyone has any input, I would very much love to hear it.

Here is a bunch of thoughts in no particular order.

• I know Renovate bot has automatic grouping in the enterprise offering but I haven't explored it deeply enough to see how the grouping mechanism works. It might be a solution to getting images from different apps into the same PR on the repo which stores the manifests. Maybe if the feature branch or PR labels on the source repo could be used as a matching key or something that trickles down to the manifest repo? I'm unaware of how the Argo Image Updater or Flux handle grouping like that.

• Github has a function where merging a PR can close issues using keywords like "resolves #xxx". I don't believe that same functionality exists for PRs to close other PRs though. If the source PR is closed or merged, how do we automatically close the manifest PR? An activity timeout might work but I feel like there should be something more efficient than just waiting around for a TTL to expire. Maybe a Github App could be developed for this?

• It might be nice to have the ArgoCD ApplicationSet Pull Request generator return a list of files and directories that are being modified. I know Github offers the diff info as a simple API endpoint so it would (hopefully) not end up as just some hack inside Argo. I admittedly did not investigate other git providers to see if this is universally available though. This could be used to build out the Application template along these lines:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: preview-environments
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - pullRequest:
      github:
        owner: your-org
        repo: kubernetes-manifests
        appSecretName: github-app-repo-creds
  template:
    metadata:
      name: 'preview-{{.number}}-{{.branch}}'
    spec:
      project: 'default'
      destination:
        name: 'in-cluster'
      sources:
      {{ range $path := .modifiedDirectories }}
      - repoURL: [email protected]:your-org/kubernetes-manifests.git
        targetRevision: "{{.branch}}"
        path: "{{ $path }}"
      {{ end }}

• For testing, it might be necessary to spin up a handful of secondary microservices but not the entire stack to test a small change. Maybe the ApplicationSet git generator could be used to determine service-level dependencies. In the directory for each application's manifest, a dependencies.json file could be defined that lists other apps that this particular one would depend on. That file could be read by the files option in the generator and used in a similar way to the above ApplicationSet template example. I haven't tested to see if it would be possible to get dependencies of dependencies though.

• A much more complicated (but maybe more fully featured?) option would be to build out a custom operator that defined all of the dependencies required for the preview environment. You could build out the dependency and tag/version logic external to the CD tool doing the final applies. Whatever you did externally would basically just need to render down to a yaml file that could be applied to the cluster. This idea was mostly borrowed from this post on medium.

• All/any of these options could be used to either apply manifests directly to a cluster with Argo/Flux or by using something like vCluster to provision a virtual cluster, pre-seed that with an Argo installation, and have an ApplicationSet inside the vCluster compile the needed resources, applying them to this new local environment. The benefit would be that developers can build cluster-level resources (operators, CRDs, etc) in a safe manner.

[hawksight]

Could we maybe define what changes that span multiple microservices means?

• What a simple example might look like? (Podinfo with redis?)
• What a complicated example might be? (Google Microservices Application)?

Lots of good ideas to look into above from @n-029894.

I'm just wondering if there's a simple solution like:

• One helm chart that has multiple dependencies deployed from an Application or ApplicationSet. So the dependencies are wrapped in helm.
• Similar to above but with a larger Kustomize abstraction?
• Application using spec.sources to define multiple sources for an application (untried atm)

I think with a concrete definition it'll be easier to just try some options and see what the best fit is. Likely different for different people :)

[n-029894]

Could we maybe define what changes that span multiple microservices means?

Yeah that's a good point. My use case is that, occasionally, there are changes that are being built or prototyped on one service but they also depend on changes to a secondary service. It would be more efficient to make those changes concurrently. Obviously the dev could manually update the branch that the PR is opened on to change the image tag of the dependent service. It feels like there should be a more automated way of doing that though.

a larger Kustomize abstraction?

That's an interesting thought. Were you thinking more about a layer of kustomize on top of the existing directory structures or adding the dependencies as additional resources into the existing kustomization.yaml files?

The later sounds like it should be feasible. It would probably (might?) need the git diff info from the PR generator to assemble the dependencies though. We would need to know what is being modified so that the resources being generated are contained exclusively within that app. Like given this directory structure:

dependency-example
├── app_1
│   ├── base
│   │   └── kustomization.yaml
│   └── overlays
│       └── ephemeral
│           ├── deployment.yaml
│           ├── kustomization.yaml
│           └── service.yaml
├── app_2
│   ├── base
│   │   └── kustomization.yaml
│   └── overlays
│       └── ephemeral
│           ├── deployment.yaml
│           ├── kustomization.yaml
│           └── service.yaml
└── app_3
    ├── base
    │   └── kustomization.yaml
    └── overlays
        └── ephemeral
            ├── deployment.yaml
            ├── kustomization.yaml
            └── service.yaml

If app_3/overlays/ephemeral/kustomization.yaml looked like this, we would need to make sure that the git generator only generated an application based on the app_3 ephemeral overlay.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - ../../app_1/overlays/ephemeral
images:
  - name: my-image
    newTag: 1.0.0

Patterns like this would not work in that situation. The matching pattern used for the path option in the git generator would need to be pretty specific to avoid also applying the resources defined in app_1 or app_2. That's where I feel having a {{ .diffDirectories }} attribute provided by the PR generator might be valuable.

generators:
- matrix:
    generators:
    - git:
        repoURL: https://github.com/your-org/kubernetes-manifests.git
        revision: HEAD
        pathParamPrefix: app
        directories:
        - path: '*'
    - git:
        repoURL: https://github.com/your-org/kubernetes-manifests.git
        revision: HEAD
        pathParamPrefix: env
        directories:
        - path: '{{app.path.basename}}/overlays/ephemeral'

@kostis-codefresh @christianh814 Either of you guys ever see this come up before?

[hawksight]

Yeah that's a good point. My use case is that, occasionally, there are changes that are being built or prototyped on one service but they also depend on changes to a secondary service. It would be more efficient to make those changes concurrently. Obviously the dev could manually update the branch that the PR is opened on to change the image tag of the dependent service. It feels like there should be a more automated way of doing that though.

This sounds like an ephemeral version of Dan's talk using Rollouts. So you might have a frontend and backend that you want to preview in tandem.

I'm going to assume that for the majority of people they would currently deploy each app on it's own. Frontend app, backend app etc.
So the goal here is if you can have a PR generated preview environment that includes changes from multiple applications?

I can't comment on your kustomization theory, I'm not the biggest user of that tooling, so hopefully others have better insight there.

[n-029894]

So the goal here is if you can have a PR generated preview environment that includes changes from multiple applications?

Yes exactly. A frontend/backend change previewed in tandem is a great example.