diff --git a/ADR/0001-pipeline-service-phase-1.md b/ADR/0001-pipeline-service-phase-1.md index 76d08f0b..733deeda 100644 --- a/ADR/0001-pipeline-service-phase-1.md +++ b/ADR/0001-pipeline-service-phase-1.md @@ -1,11 +1,13 @@ # 1. Pipeline Service Phase 1 Created: 2022-10-13 -Last Updated: 2022-10-13 +Last Updated: 2023-09-29 ## Status -Accepted +Obsolete + +Superceded by [ADR-0009](./0009-pipeline-service-via-operator.md) ## Context diff --git a/ADR/0009-pipeline-service-via-operator.md b/ADR/0009-pipeline-service-via-operator.md new file mode 100644 index 00000000..bdbf5400 --- /dev/null +++ b/ADR/0009-pipeline-service-via-operator.md @@ -0,0 +1,74 @@ +# 0009 Pipeline Service via Operator + +Created: 2023-09-29 +Last Updated: 2023-09-29 + +## Status + +Proposed + +## Context + +kcp is no longer being used as a control plane for RHTAP. This means that +"Pipeline Service" cannot be deployed as an independent service. For our +initial MVP, all Tekton APIs need to be deployed onto a standard OpenShift +cluster (specifically OpenShift Dedicated). + +## Decision + +All Tekton APIs will be provided using the stock OpenShift Pipelines operator (OSP). +In the spirit of developing in a "Service First" manner, RHTAP will deploy +a candidate "nightly" release of the operator. The service will be defined in +the [pipeline-service](https://github.com/openshift/pipeline-service) +repository, which is then imported into +[infra-deployments](https://github.com/redhat-appstudio/infra-deployments) as +an ArgoCD application. + +Not all metrics required for operating the service are exposed natively by the +controllers. The `pipeline-metrics-exporter` controller is to be used as a test +bed to expose new metrics, with the goal of upstreaming those metrics as they +mature and prove their value. + +Configurations that are specific to RHTAP must be made available through +the OSP CRDs. The following changes are specific to RHTAP: + +- Disable Tekton Triggers. Pipelines as Code will be the favored mechanism for + event-based triggering of pipelines for now. This decision can be revisited + in the future based on need. +- Disable the pruner that comes with the operator. Tekton Results will be used + to prune `PipelineRun` and `TaskRun` data off cluster, thereby ensuring data + is archived before removal. +- Direct Pipelines as Code to use a URL pattern that displays the `PipelineRun` + or `TaskRun` info from the Hybrid Application Console (HAC). This ensures + end users do not need access to the underlying compute cluster(s). +- The Pipelines as Code application name must match the GitHub Application name, so that users understand which GitHubApplication is responsible for triggering the pipelines. +- The GitHub Application secret value, deployed using an ExternalSecret. +- Any configuration related to performance. + +Furthermore, as the service will be accessed through CodeReadyToolchain (CRT), the +following changes are also specific to RHTAP: +- Deploying a proxy (known as `SprayProxy`) on the CRT host cluster that redirects + incoming PaC requests to the member clusters. Requests are redirected only if: + - the payload is validated using the webhook secret; + - the payload size is under 25MB. +- Providing a plugin to the CRT Proxy so Tekton Results requests are redirected + to the appropriate member cluster. + +## Consequences + +- Tekton Triggers should be disabled in RHTAP using the appropriate operator + configuration. +- The Tekton Pruner needs to be disabled in RHTAP using the appropriate + operator configuration. This is done under the assumption that Results will + be responsible for pruning resources. Eventually the operator should automate + this setting if `Results` is deployed and configured to prune resources. +- Pipelines as Code should use an appropriate URL to HAC when interacting with + SCM services, such as the GitHub + [Checks API](https://docs.github.com/en/rest/guides/getting-started-with-the-checks-api?apiVersion=2022-11-28). +- Changes to Pipeline components need to be baked into the operator and built + rapidly. +- Hot fixes need to be provided on a "roll forward" basis. OLM Operators do not + support "rollback" mechanisms today. Mean time to revert an offending change, + rebuild, and deploy needs to be measured in hours. +- The version of the deployed operator needs to be configurable via ArgoCD. + This is should be doable by using `kustomize` to patch the `CatalogSource` and `ImageContentSourcePolicy`. diff --git a/book/pipeline-service.md b/book/pipeline-service.md index 5ddc7af3..5a355db5 100644 --- a/book/pipeline-service.md +++ b/book/pipeline-service.md @@ -1,28 +1,50 @@ # Pipeline Service -Pipeline Service provides a SaaS for pipelines. It leverages: +Pipeline Service provides Tekton APIs and services to RHTAP. +In the initial phase of RHTAP, Pipeline Service will be provided by a stock +installation of the OpenShift Pipelines operator. +This deployed version will be the a candidate build of the OpenShift Pipelines +operator from a Red Hat build system. -- OpenShift for the compute. -- Tekton Pipelines, Chains, Pipelines as Code, Results for the core of the service. -- OpenShift GitOps / Argo CD for managing the infrastructure. +![Pipelines operator deployment](../diagrams/pipeline-service.drawio.svg) -The following features are provided: +## APIs and Services -- Signing and attestation of `TaskRuns` with [Tekton Chains](https://tekton.dev/docs/chains/). -- Archiving/Pruning of `PipelineRuns` and `TaskRuns` with [Tekton Results](https://tekton.dev/docs/results/) (not activated). -- Integrations with GitHub, Gitlab, and Bitbucket with [Pipelines as Code](https://pipelinesascode.com) (PaC). +Pipeline Service provides the following: + +- Tekton APIs directly through its custom resource definitions. +- Container image signing and provenance attestations through Tekton Chains. +- Archiving of `PipelineRuns`, `TaskRuns`, and associated logs through Tekton + Results. + +Pipeline Service also exposes the following ingress points: + +- Pipelines as Code controller: this is a `Route` that receives webhook events + from source code repositories. +- Tekton Results API: this is an `Ingress` that serves Tekton Results data + over a RESTful API. Clients authenticate with the same `Bearer` token used to + authenticate Kubernetes requests. + +## Deployment Configuration + +The deployment of the OpenShift Pipelines operator will have the following +notable configurations: + +- Tekton Triggers will be disabled entirely. +- The pruner (provided by the Pipelines operator) will be disabled in favor of + pruning via Tekton Results. +- Pipelines as Code will link the results of pipeline tasks to an appropriate + [Hybrid Application Console (HAC)](./hybrid-application-console.md) URL. ## Architecture -### Component +### Diagram -The Pipeline Service component is deployed onto the cluster alongside other AppStudio components. The following forms the core architecture of Pipeline Service: +Legend: +* Blue: managed by Pipeline Service +* Yellow: not managed by Pipeline Service -1. _OpenShift Pipelines (OSP):_ It enables the creation and management of Pipelines for user [Applications](https://redhat-appstudio.github.io/book/ref/application-environment-api.html#application). -2. _Pipelines as Code (PaC):_ An OpenShift Pipelines component that enables receiving webhook events from version control systems. The user would have to install a [GitHub App](https://pipelinesascode.com/docs/install/github_apps/) with the required permissions in order to forward events to the PaC endpoint exposed by the platform. -3. _Spray Proxy:_ Forwards requests to all clusters in the environment once an event is received on PaC. It is a reverse proxy that broadcasts to multiple backends. -4. _Tekton Results:_ Responsible for storing logs and pruning PipelineRuns. Utilises AWS RDS as the backend database for our production instance. -5. _Tekton Chains:_ Provides PipelineRun attestation capabilities, ensuring that all PipelineRuns are verified and can be traced back to their source. +![Architecture diagram](../diagrams/pipeline-service/architecture.jpg) ### appstudio-pipeline Service Account @@ -35,4 +57,4 @@ The [CodeReadyToolchain](https://github.com/codeready-toolchain) platform (CRT) ## Repository -The official repository for the Pipeline Service can be found at https://github.com/openshift-pipelines/pipeline-service. This repository contains the source code, configuration files, and documentation needed to deploy and consume the service. \ No newline at end of file +The official repository for the Pipeline Service can be found at https://github.com/openshift-pipelines/pipeline-service. This repository contains the source code, configuration files, and documentation needed to deploy and consume the service. diff --git a/diagrams/pipeline-service/architecture.jpg b/diagrams/pipeline-service/architecture.jpg new file mode 100644 index 00000000..c5a6e098 Binary files /dev/null and b/diagrams/pipeline-service/architecture.jpg differ