Lifecycle Management with GitOps

[williamcaban]

Lifecycle Management with GitOps

The intent here is to have a discussion about how the Lifecycle Management (LCM) cycle of of an operation of infrastructure, platform, evironments or applications and eventually bring it as an official document either as best practice, white paper or documentation of the patterns.

Proposed definitions for terms used in the discussion:

• Infrastructure: physical and virtual resources and the related ancillary services required by the platform and services
• Platform: software framework from that host applicatons or services
• Environment: conceptual grouping of resources across one or more platforms used to run appllications
• GitOps Controller: system capable of following the GitOps principles pulling the immutable version of the desired state and continuously reconciliating over a domain of action

Discussion

For the purpose of the discussion I use the term Lifecycle Management (LCM) as series of stages:

• Release Commit new version
• Test
• Deploy
• Operate
• Monitor & Analyze
• Optimize

Note: There could be more or there could be fewer stages. That should be part of the discussion.

How does these stages map to GitOps?

Release Commit New Version

The release of a A "new version" of the infrastructure configuration, platform or application starts with a Pull Request (PR) commit to the GitOps repository

Test

• When a PR is submitted, a pipeline should be executed to do functional testing and end-to-end integration testing for infrastructure, platform and application before allowing the PR for consideration
• Once the PR has been validated by pipelines, approvers must approve the change before the pipeline merge changes to the GitOps repository
  • If a change can only happen at specific time (e.g. maintenance window) or should not happen during specific time (e.g. blackout time), and the concept is not understood by the artifact being reconciled, the pipeline should not proceed with the merge until the approve time range is met.

Deploy

• When a PR is merged into the GitOps repository, After a commit to the GitOps repository, the GitOps controllers ~~will~ follows the continuous remediation cycles (principles 4 and 5) to apply the changes to the systems
• A rollback is a redeployment of a previous version by submitting a new PR commit for it (e.g. git revert, git reset, or similar mechanisms)
  • e.g. A rollback initiate a “Release New Version” stage commit with a previous version

Operate

• When any modification for adjustment is required, the operations team or tools acting for them (e.g. AIOps tools), generate a PR with the desired modifications
• The PR triggers a new “Release New Version” stage and pipeline
• Git is THE interface for Operations

Monitor & Analyze

• There is no direct mapping of this stage to a GitOps construct. This is part of the reconciliation cycle of an outer GitOps controller (A GitOps controller managing other GitOps controller)

Optimization

• When any modification for adjustment is required, the operations team or tools acting for them (e.g. AIOps tools), generate a PR commit with the desired modifications
• The PR commit triggers a the "New version" “Release New Version” stage and pipeline

[lloydchang]

@williamcaban wrote:

Note: There could be more or there could be fewer stages. That should be part of the discussion.

Fewer stages is better.

To simply Lifecycle Management, I recommend removing Release New Version and Test stages because they:

1. Will vary greatly in practice
2. Can happen outside of the scope of GitOps
3. Are better in the scope of Continuous Integration (CI), which is outside of the scope of GitOps
4. Don't directly map to GitOps, which focuses on Continuous Delivery (CD) and Continuous Operations (CO)

There should be a loose coupling between CI and CD to accomplish GitOps (CD + CO).

To illustrate what I mean:

Release new version: The release of a "new version" of the infrastructure configuration, platform or application starts with a Pull Request (PR) to the GitOps repository

Not always. For example, https://fluxcd.io/docs/guides/image-update/

Test: When a PR is submitted, a pipeline should be executed to do functional testing and end-to-end integration testing for infrastructure, platform and application before allowing the PR for consideration

Not always. For example, https://github.com/gitops-working-group/gitops-working-group/discussions/56

Thank you for the discussion.

[williamcaban]

• I agree the first init to a repository is not a PR. How would you express the fact that the initial state of the system needs to be described as the desired state and also be versioned? Should we call it "Initial version or New version"
• A "test" is an integral part of any LCM even when it looks different for the different domains of actions (e.g. infrastructure vs software). For example, an operations team needs to validate/test a change to a system before applying it to production. With GitOps focused in the Operation aspects of a system, how can we describe that here? Instead of removing it, I rather work on figuring the right wording to describe the intent behind the stage in a way that is applicable across domains.

[lloydchang]

Not all GitOps changes require a PR. For example, https://fluxcd.io/docs/guides/image-update/ doesn't require a PR; image reflector controller and image automation controller's GitOps changes are rules-based and fully automated without any PRs at all.

Versioning is built into Git. I wouldn't worry about "initial" or "new", which are minor details. Each Git commit has a unique SHA hash across all Git repositories in the Universe.

When our GitOps Principle says Versioning, I interpret it as an abstraction that is implemented as Git SHA hashes. I believe most people find Versioning as an acceptable conventional English word to summarize Git SHA hashes. Optionally, one can use Git tags to create semantic release versions to make it easier to map a Git SHA hash to human-friendly version numbers.

Since Versioning is built into Git, and PRs are optional for GitOps, I still recommend removing the "Release" stage from Lifecycle Management. Otherwise, it feels like we're just reinventing the wheel that Git has already created.

Testing: The type of test that GitOps would typically use are unit test driven, such as OPA conftest — If we need the right wording, then I would look to https://www.conftest.dev/ which is part of OPA, a graduated CNCF project.

[williamcaban]

• Doing a PR or a direct commit, is more an implementation detail. Let's call it a commit.
• WRT to testing, I agree about the "unit test" level. We can provider linters and policy validation tools as examples.

Thank @lloydchang for all the comments.

[lloydchang]

@williamcaban I suggest the following changes:

---

Change from:

Release New Version
The release of a "new version" of the infrastructure configuration, platform or application starts with a Pull Request (PR) to the GitOps repository

To:

Create

• git commit to create a SHA in a GitOps repository

---

Change from:

Test

• When a PR is submitted, a pipeline should be executed to do functional testing and end-to-end integration testing for infrastructure, platform and application before allowing the PR for consideration
• Once the PR has been validated by pipelines, approvers must approve the change before the pipeline merge changes to the GitOps repository
  • If a change can only happen at specific time (e.g. maintenance window) or should not happen during specific time (e.g. blackout time), and the concept is not understood by the artifact being reconciled, the pipeline should not proceed with the merge until the approve time range is met.

To:

Test

• kustomize build ~/someApp|conftest test -

---

In the above formal example, I didn't include kubeval as a validation test because kubeval isn't in a CNCF project. That being said, you can use conftest and kubeval in combination to validate:

Informally, here is an example from https://github.com/stefanprodan/kube-tools

       echo "Run conftest"
       kustomize build test/kustomize | conftest test -p test/policy -
       echo "Run kubeval"
       helmv3 template ./charts/test | kubeval --strict

Informally, there are validation tools summarized at https://kubevious.io/blog/post/top-kubernetes-yaml-validation-tools

• Please note that since Kubevious published the following, the pros and cons may be biased?

Tool	Pros	Cons
conftest	Robust Rego based framework for custom checks/validations. Policy sharing via OCI.	Does not come with built-in checks. Steep learning curve
kubeval	YAML validation against a specific version of API Schema.	Custom resources definitions (CRD) cannot be used to add custom checks to the Kubernetes cluster.
copper	Javascript-based framework for writing custom YAML validation rules	No inbuilt rules. Users have to create custom checks.
kubevious	Visual overview of configurations, errors, best practice violations. Customizable.	Has to be deployed on every K8s cluster. But there is a Kubevious Portable edition that doesn’t require installation
config-lint	Supports for various configuration formats (Terraform, YAML, JSON) for writing custom checks.	Not sufficient built-in rules for standard checks.
Kube Linter	Powerful for out of the box security validation	Early-stage project. Alpha release
kube-score	Does YAML validation against a variety of security best practices.	No specific API version support.
Kube library	Implements an Extensible Robot Framework for testing. Python for writing custom checks.	Learning curve with Keywords based custom checks.

[williamcaban]

I've updated the original text taking into consideration the discussions.