Using Hera and ArgoCD together

[menzenski]

We adopted Argo Workflows for our ELT processes at the beginning of the year.

We're defining those workflows in Python using Hera, and these Python files live in our ELT repository alongside the definitions of the jobs that they invoke. All of our Kubernetes resources are managed with ArgoCD - I've set up a process to get the Hera-defined workflows into our Kubernetes clusters via ArgoCD, but it's not a great process as it's got some manual steps, and I don't want that.

I'm interested to connect with anyone else who's using both Hera to define workflows and a GitOps tool like ArgoCD or Flux to deploy those workflows. I'll be interested to know what other people are doing in this area - I'd like to build a more automated solution for our needs and am trying to do some research before I begin that work.

Here's an example of what we're doing:

We define each Argo Workflows resource (CronWorkflow or WorkflowTemplate) in its own Python file. Each file has an if __name__ == "__main__": block that prints the YAML manifest for that resource (using the Hera to_yaml method) when that Python script is invoked directly.

# filename: my_cron_workflow.py

from hera.workflows import CronWorkflow
from hera.workflows.models import WorkflowTemplateRef

with CronWorkflow(
    name="my-cron-workflow",
    schedule="0 7,19 * * *",
    timezone="America/Chicago",
    concurrency_policy="Replace",
    image_pull_secrets=["my-secret"],
    service_account_name="my-service-account",
    workflow_template_ref=WorkflowTemplateRef(name="my-workflow-template"),
) as cw:
    pass

if __name__ == "__main__":
    print(cw.to_yaml())

These Python scripts are located in an argo/src/ directory in our git repository. We have a helper script defined (we are using PDM to manage Python dependencies, these scripts are defined as PDM shell scripts) to generate the YAML manifests in a gitignored argo/out/ directory:

Build the workflows:

mkdir -p argo/src; mkdir -p argo/out; for w in argo/src/*.py; do b="$(basename $w)"; n="${b%.*}"; python "$w" > "argo/out/$n.yaml"; done

The repository where these Hera scripts live defines a GitHub Actions job that runs on pull requests - it generates the YAML from all the scripts and validates the resulting Kubernetes manifests against the Argo Workflows schemas using kubeconform. Aside from the obvious benefits of validating against the schemas, this check will error and fail if one of the Python scripts does not define that if __name__ == "__main__": block. So it helps enforce that convention as well:

  validate_argo_resource_definitions:
    runs-on: ubuntu-latest
    needs:
      - pre-commit
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up PDM
        uses: pdm-project/setup-pdm@main
        with:
          python-version: 3.11.1
      - name: Install Argo Workflows dependencies
        run: |
          pdm sync -d -G argo
      - name: Build YAML resource manifests
        run: |
          pdm run build-workflows
      - name: Login to Github container registry
        run: echo "${{ github.token }}" | docker login https://ghcr.io -u ${GITHUB_ACTOR} --password-stdin
      - name: Pull kubeconform Docker image
        run: docker pull ghcr.io/yannh/kubeconform:master
      - name: Create kubeconform container
        run: docker create ghcr.io/yannh/kubeconform:master
      - name: Run kubeconform linter
        id: kubeconform-linter
        run: |
          docker run \
            --workdir /github/workspace \
            -v "/home/runner/work/my-repo/my-repo/argo":"/github/workspace" \
            ghcr.io/yannh/kubeconform:master \
            -schema-location default \
            -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/argoproj.io/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
            -summary -output json /github/workspace/out > result.json
      - name: format
        if: always()
        run: |
          echo "result<<EOF" >> $GITHUB_ENV
          echo "$(cat result.json)">> $GITHUB_ENV
          echo "EOF" >> $GITHUB_ENV
      - name: Comment PR
        if: always()
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: kubeconform-result-comment
          message: |-
            ### Kubeconform Validation Results
            ```json
            ${{ env.result }}
            ```

This is where the process becomes manual 😬

After a PR is merged, I pull the latest main branch of this repository, and I run the pdm run build-workflows command to generate all the YAML manifests. Then, I copy them all into a separate repository where our ArgoCD-managed Argo Workflows manifests are located. We are running Argo Workflows in six environments (six Kubernetes clusters), so I copy the output files to each environment separately:

# build YAML manifests
$ pdm run build-workflows

# copy YAML manifests to dev environment ArgoCD directory
$ cp argo/out/cron_* ../my-argocd-repository/dev/cron-workflows/
$ cp argo/out/template_* ../my-argocd-repository/dev/workflow-templates/

# copy YAML manifests to staging environment ArgoCD directory
$ cp argo/out/cron_* ../my-argocd-repository/staging/cron-workflows/
$ cp argo/out/template_* ../my-argocd-repository/staging/workflow-templates/

# repeat the above for our other four environments

We also have some per-environment complications (for example, some CronWorkflows shouldn't be defined in all environments), so after I run those copy commands, I do that cleanup manually.

Then I open a pull request in that ArgoCD-managed repository, and when that pull request is merged, ArgoCD gets my changes out into the appropriate clusters.

I think there are several opportunities to improve our current process:

1. Expanding our "build script" from a bash one-liner into something more parameterized (probably Python). This would allow us to generate the YAML manifests in an environment-aware way.
2. Automating the creation of the changes in the ArgoCD-managed repository. This should be achievable with GitHub Actions - we could have the action make a git commit to a branch and open a pull request, or potentially the action could just commit directly to main in that other repository.

Is anyone else using Hera with ArgoCD? What's your process look like?

[Joibel]

I've built a version of lovely plugin which will take .py files from git and using ArgoCD deploy them to your cluster.

It's barely tested, as I'm not a hera user. To give it a go in ArgoCD install the [ghcr.io/crumbhole/lovely-hera:0.19.1] container as a sidecar plugin to ArgoCD, following the guide here.

Put some hera python in a git directory with obj.toFile('.') for each workflow-template or cron-workflow, and create an argocd application pointing at that git directory, and with plugin name lovely-hera. My guide is here.

It should take the directory and render all the objects and put them into your cluster for you.

I'm happy to help with wherever you get to doing this and make this a bit more formal. PRs also welcome!

[frank-bee]

Hi @menzenski ,

I just started to develop a blueprint hera repository which can be used by our teams.
Deployment is also done via gitops, flux in our case but with argo this should also work
See here https://github.com/aiknow-public/hera-blueprint

The basic idea is very simple: The workflows of the project are rendered as yamls (WorkflowTemplates actually) by a github pipeline.
And flux / argo pulls for the changes (basically listens to any modification in the kubernetes folder.

Would something like this work for you as well?
We are still very new to hera / argo, so really interested in exchanging ideas here...