From 522699bc5b66763645f49e11dba237be16b9aa29 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Fri, 25 Aug 2023 13:42:06 +0200 Subject: [PATCH 01/21] Add zuul user documentation draft Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd.md | 201 ++++++++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 operations/operations/zuul-ci-cd.md diff --git a/operations/operations/zuul-ci-cd.md b/operations/operations/zuul-ci-cd.md new file mode 100644 index 0000000000..4cae084120 --- /dev/null +++ b/operations/operations/zuul-ci-cd.md @@ -0,0 +1,201 @@ +# Zuul users guide + +## Prerequisites + +1. Repository is known by your zuul instance +2. Basic ansible knowledge +3. Basic yaml knowledge +4. zuul-client installed + +## Who is it for? + +You may have heard about zuul and may ask yourself if it is capable to support you. +Basically everything you use ansible for can be done using zuul. That is not always +a good thing since you may get careless and your workload will exceed the CI/CD concept. + +If you find yourself doing things under the following list you are at the right place. + +1. Code testing +2. Deployment tests using IaC + +If you want to, let's say, monitor something using zuul, that is possible but not the +intended use case. + +## Where do I start? + +Right in your projects repository! The only prerequisite you need to fulfill, is that +your repository you want zuul to work on is known by zuul. This is done by the zuuls +tenant configuration. To update this configuration you need access to the zuul instance +or ask an administrator for help. + +We consider that zuul knows about your repository so we can get started. There are three +topics that you should know about. To get jobs running you need the "job" itself. Jobs run +within "pipeline". The third important thing is to provide a "project" definition. + +### Projects + +If zuul is configured to observe your repository it will have a look at your projects +definition. Minimal example: + +``` +--- +- project: + name: my-org/my-repo + default-branch: main + merge-mode: "squash-merge" + my_pipeline1: + jobs: + - my_job1 + - my_job2 + ...... + my_pipeline2: + jobs: + - my_jobs + ... + +``` +By default zuul will observe all branches for such files. We have to set the repository name +that have to match the exact value that was set for zuul. Set a default-branch where actions +that don't match an explicit branch are executed on. Set the merge-mode that zuul has to use. +But beware that not all issue tracker support all methods. For github squash-merge will work. + +After these three properties add the pipelines you want to use to the project definition. +With the `jobs` list you define which jobs to run in which pipeline. + + +### Pipelines + +Every zuul instance will have at least one repository that is used for configuration. There +you will find the available pipelines. Pipelines are used to run your jobs on a periodic or +event driven base. Pipelines can be used to run other pipelines and to keep your jobs in a +defined order if you need this. + +Have a look at the configuration repository to utilize the pipelines for your repository. You +are not able to define new pipelines outside of a so called "configuration" repository. Since, +by default your repo is considered "untrusted". So in the first place you don't need to +think about, how to create a pipeline. Just use one that fits your needs as close as possible. + +### Jobs + +All jobs that your zuul instances knows of can be used for your own purposes. +Call them directly or implement a job that uses an existing job as parent. +Didn't find the right job? Than we have to create a new one. Existing jobs +can be found in the web ui of your zuul instance: https://zuul.scs.community/t/SCS/jobs + +First have a look on a basic job example: + +``` +- job: + name: base + parent: null + description: | + The recommended base job. + + All jobs ultimately inherit from this. It runs a pre-playbook + which copies all of the job's prepared git repos on to all of + the nodes in the nodeset. + + It also sets a default timeout value (which may be overidden). + pre-run: playbooks/base/pre.yaml + post-run: + - playbooks/base/post.yaml + - playbooks/base/post-logs.yaml + roles: + - zuul: zuul/zuul-jobs + timeout: 1800 + nodeset: + nodes: + - name: ubuntu-jammy + label: ubuntu-jammy + +``` +Each jobs needs a name. Each job need to define whether there is parent job or not. +Jobs without a parent are called "base" jobs. Usually you don't want to implement base jobs since +there are already some base jobs that implement often used stuff. A description may not be mandatory +but is obviously useful. + +Necessary for zuul to do anything you just need to add a `run` or `roles` property. Within something that is +like a `noop` job or just printing something to stdout that is everything you need to run your first job. +Since anything we want to do requires a little bit more you have to define a nodeset. The nodes +are used to run your playbooks on. In 99,9% you will need this too. + +The properties `pre-run` and `post-run` are useful for bootstrap and cleanup. If your actual job wants to create +bootstrap some infrastructure you can to this in the `pre-run`. Using an cloud provider you want to release +no longer used resources. That can be done in the `post-run`. If you are using a parent job it is likely +that the parent job may has pre- and post-run playbooks. In this case your pre- and post-run playbooks are +"nested". Example: + +1. pre-run parent +2. pre-run my job +3. post-run my job +4. post-run parent + +If your job exceeds the defined timeout, the job is considered as failed. + +#### What about secrets? + +Right now you should be able to run basic tasks. But what if you try to test something +that needs credentials to connect to an outside service? Or you have to address additional +ressources in an openstack environment and you have to use something like app credentials? + +That is where job secrets are used. Example: + +``` +- job: + name: SOME_JOB + parent: base + description: | + A job basic job used as example + secrets: + - name: clouds_conf + secret: app_credential_cloud_conf + run: playbooks/my-playbook.yaml +``` + +Secrets for a job are simply defined by the keyword `secrets`. +Each secret needs a name that can be used in your playbooks. +The property `secret` references the secret that is defined within your project. + +Example: + +``` +- secret: + name: app_credential_cloud_conf + data: + credentials: my-secret-value +``` +Within `my-playbook.yaml` you can reference the secret value using `"{{ clouds_conf.credentials }}"`. +In this example `my-secret-value` is clear readable text. That is not something we want to keep +secrets. But how do you encrypt secrets in a way that they are secure and also can be decrypted by +zuul? + +For this purpose zuul creates its own public/private key pair for each project. Everyone may use the +public key to create secrets. But only zuul will be able to decrypt these values. To avoid the user +to be responsible for the correct encryption there is an zuul-client tool that will do this for you. + +Example: +``` +zuul-client --zuul-url ZUUL_URL encrypt --tenant TENANT --project ORGANIZATION/PROJECT --infile creds.yaml --outfile clouds.yaml.enc +``` + +The content may look like this: +``` +- secret: + name: + data: + : !encrypted/pkcs1-oaep + - IGZ2Wu47R9mEY4fjetbxSAUGNaz4HR1mjk9lCLq3HsUMjHGj9YPlb2MvnPQw1LCJSvpaK + ogth7hi2zYwrs5tNAik/qlVSB7AM+LQRP7lmlM4JmD6WOyR7DisHu7oMD1Gqem2ZuMggA + DIBn5+DeBIvnwihDOcS+BKPTVMEtXOJNkuObZHE8DweB/RQIGUvjyeq5yoAmz/y+qGVqe + 0Vk4pTYFIBgk5DMzwVnDzDkqs/QokoOupMUoBcpapmM11do4ymjbDpeINjayoro6VXTtX + Mkk9fDv9wuJIQTuyHAOfMD+UYS/HqVSF/Hm9ScUvfhw02gTdzKCxliWhFHJOj7RbdUUMK + OYYcUkNp5cXZUYFnflMhxVEnzREbdAIklNPfoHOizsxLPaUZ9yk6XcFRflFfMvqBtUS00 + LCx0Uh906NwdaEUrv2ZdrN123rwfwfw4333232rDFDFfsdfddsfdDFSFSdqrrtwms5Mi0 + szUBaM4j+Mayep+41vl0cpsLU91GzXEATWMaPIN8OnEHF6qQIv0wB6VaKd5aeAyERisb3 + wFdjEo4faLO70RWzR33k+4xqAYNIIFyTMpWJz21CUSfoYG8ygL6t7RJGgyjA+0KsVEyj+ + ewEtiaUOLYyD7pXtqdw1HgzjqiXnfxk+wSv/y5y/TGGYpQj8zU76jS7Zj0ft/0= +``` + +You may use this content or the file to provide it as a secret. You just have to update the `` and the +`` part. + From ab0c8f9acc72a7e202080ed3be9f1bf52d3f08f7 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Fri, 25 Aug 2023 13:45:49 +0200 Subject: [PATCH 02/21] add parantheses Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/operations/operations/zuul-ci-cd.md b/operations/operations/zuul-ci-cd.md index 4cae084120..5bd27f42a5 100644 --- a/operations/operations/zuul-ci-cd.md +++ b/operations/operations/zuul-ci-cd.md @@ -80,7 +80,7 @@ think about, how to create a pipeline. Just use one that fits your needs as clos All jobs that your zuul instances knows of can be used for your own purposes. Call them directly or implement a job that uses an existing job as parent. Didn't find the right job? Than we have to create a new one. Existing jobs -can be found in the web ui of your zuul instance: https://zuul.scs.community/t/SCS/jobs +can be found in the web ui of your zuul instance: (https://zuul.scs.community/t/SCS/jobs) First have a look on a basic job example: From 1b3996fd390b53251c68be9195d7482852fd27c9 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Fri, 25 Aug 2023 13:47:09 +0200 Subject: [PATCH 03/21] add text for the link Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/operations/operations/zuul-ci-cd.md b/operations/operations/zuul-ci-cd.md index 5bd27f42a5..42ca02b8bb 100644 --- a/operations/operations/zuul-ci-cd.md +++ b/operations/operations/zuul-ci-cd.md @@ -80,7 +80,7 @@ think about, how to create a pipeline. Just use one that fits your needs as clos All jobs that your zuul instances knows of can be used for your own purposes. Call them directly or implement a job that uses an existing job as parent. Didn't find the right job? Than we have to create a new one. Existing jobs -can be found in the web ui of your zuul instance: (https://zuul.scs.community/t/SCS/jobs) +can be found in the web ui of your zuul instance: [Example](https://zuul.scs.community/t/SCS/jobs) First have a look on a basic job example: From ed06e4569b60d0ea75ee2c8c2668350db5685948 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Fri, 25 Aug 2023 13:51:40 +0200 Subject: [PATCH 04/21] fenced code blocks do have a type now Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/operations/operations/zuul-ci-cd.md b/operations/operations/zuul-ci-cd.md index 42ca02b8bb..a83dedd317 100644 --- a/operations/operations/zuul-ci-cd.md +++ b/operations/operations/zuul-ci-cd.md @@ -37,8 +37,7 @@ within "pipeline". The third important thing is to provide a "project" definitio If zuul is configured to observe your repository it will have a look at your projects definition. Minimal example: -``` ---- +```yaml - project: name: my-org/my-repo default-branch: main @@ -84,7 +83,7 @@ can be found in the web ui of your zuul instance: [Example](https://zuul.scs.com First have a look on a basic job example: -``` +```yaml - job: name: base parent: null @@ -140,7 +139,7 @@ ressources in an openstack environment and you have to use something like app cr That is where job secrets are used. Example: -``` +```yaml - job: name: SOME_JOB parent: base @@ -158,7 +157,7 @@ The property `secret` references the secret that is defined within your project. Example: -``` +```yaml - secret: name: app_credential_cloud_conf data: @@ -174,12 +173,12 @@ public key to create secrets. But only zuul will be able to decrypt these values to be responsible for the correct encryption there is an zuul-client tool that will do this for you. Example: -``` +```bash zuul-client --zuul-url ZUUL_URL encrypt --tenant TENANT --project ORGANIZATION/PROJECT --infile creds.yaml --outfile clouds.yaml.enc ``` The content may look like this: -``` +```yaml - secret: name: data: From f1aa9ad84dbf68cec92c9685d2bd1e830feb2aa5 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Fri, 25 Aug 2023 13:54:43 +0200 Subject: [PATCH 05/21] add blanks to jusity check Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/operations/operations/zuul-ci-cd.md b/operations/operations/zuul-ci-cd.md index a83dedd317..aa5395c927 100644 --- a/operations/operations/zuul-ci-cd.md +++ b/operations/operations/zuul-ci-cd.md @@ -53,6 +53,7 @@ definition. Minimal example: ... ``` + By default zuul will observe all branches for such files. We have to set the repository name that have to match the exact value that was set for zuul. Set a default-branch where actions that don't match an explicit branch are executed on. Set the merge-mode that zuul has to use. @@ -108,6 +109,7 @@ First have a look on a basic job example: label: ubuntu-jammy ``` + Each jobs needs a name. Each job need to define whether there is parent job or not. Jobs without a parent are called "base" jobs. Usually you don't want to implement base jobs since there are already some base jobs that implement often used stuff. A description may not be mandatory @@ -163,6 +165,7 @@ Example: data: credentials: my-secret-value ``` + Within `my-playbook.yaml` you can reference the secret value using `"{{ clouds_conf.credentials }}"`. In this example `my-secret-value` is clear readable text. That is not something we want to keep secrets. But how do you encrypt secrets in a way that they are secure and also can be decrypted by @@ -173,11 +176,13 @@ public key to create secrets. But only zuul will be able to decrypt these values to be responsible for the correct encryption there is an zuul-client tool that will do this for you. Example: + ```bash zuul-client --zuul-url ZUUL_URL encrypt --tenant TENANT --project ORGANIZATION/PROJECT --infile creds.yaml --outfile clouds.yaml.enc ``` The content may look like this: + ```yaml - secret: name: From 7f76772d7435c5e014b900b1b825949a402c5a79 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Fri, 25 Aug 2023 13:56:46 +0200 Subject: [PATCH 06/21] add/remove blanks/whitespaces to jusity check Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/operations/operations/zuul-ci-cd.md b/operations/operations/zuul-ci-cd.md index aa5395c927..77ff64d5f4 100644 --- a/operations/operations/zuul-ci-cd.md +++ b/operations/operations/zuul-ci-cd.md @@ -62,7 +62,6 @@ But beware that not all issue tracker support all methods. For github squash-mer After these three properties add the pipelines you want to use to the project definition. With the `jobs` list you define which jobs to run in which pipeline. - ### Pipelines Every zuul instance will have at least one repository that is used for configuration. There @@ -120,7 +119,7 @@ like a `noop` job or just printing something to stdout that is everything you ne Since anything we want to do requires a little bit more you have to define a nodeset. The nodes are used to run your playbooks on. In 99,9% you will need this too. -The properties `pre-run` and `post-run` are useful for bootstrap and cleanup. If your actual job wants to create +The properties `pre-run` and `post-run` are useful for bootstrap and cleanup. If your actual job wants to create bootstrap some infrastructure you can to this in the `pre-run`. Using an cloud provider you want to release no longer used resources. That can be done in the `post-run`. If you are using a parent job it is likely that the parent job may has pre- and post-run playbooks. In this case your pre- and post-run playbooks are @@ -202,4 +201,3 @@ The content may look like this: You may use this content or the file to provide it as a secret. You just have to update the `` and the `` part. - From a5d95c606cf435dbb763209d3e6e9f6e18afeb16 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Fri, 25 Aug 2023 13:59:19 +0200 Subject: [PATCH 07/21] update filename to see its purpose Signed-off-by: Benjamin Zapiec --- .../{zuul-ci-cd.md => zuul-ci-cd-quickstart-user-guide.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename operations/operations/{zuul-ci-cd.md => zuul-ci-cd-quickstart-user-guide.md} (100%) diff --git a/operations/operations/zuul-ci-cd.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md similarity index 100% rename from operations/operations/zuul-ci-cd.md rename to operations/operations/zuul-ci-cd-quickstart-user-guide.md From baeebd7eff6d53fe713ca71dc1458d001419a7e0 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Tue, 29 Aug 2023 09:08:33 +0200 Subject: [PATCH 08/21] add an example of where to store the zuul configuration add a link to check the known projects explicit use scs zuul Signed-off-by: Benjamin Zapiec --- .../zuul-ci-cd-quickstart-user-guide.md | 28 ++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index 77ff64d5f4..4f75311691 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -2,11 +2,15 @@ ## Prerequisites -1. Repository is known by your zuul instance +1. Repository is known by [SCS zuul](https://zuul.scs.community) 2. Basic ansible knowledge 3. Basic yaml knowledge 4. zuul-client installed +Check [SCS zuul projects](https://zuul.scs.community/t/SCS/projects) for your repository to +be available. If it is missing you need an administrator to get your repository +configured to zuul. + ## Who is it for? You may have heard about zuul and may ask yourself if it is capable to support you. @@ -32,6 +36,28 @@ We consider that zuul knows about your repository so we can get started. There a topics that you should know about. To get jobs running you need the "job" itself. Jobs run within "pipeline". The third important thing is to provide a "project" definition. +## Where to save the zuul relavant data? + +Zuul will parse all branches of the untrusted repositories that zuul knows about. +Your repository is most likley an untrusted one since only the configuration repostiories should +have the "trusted" state. +So it doesn't matter whether you have just one branch containing zuul files or all branches. Zuul +is looking for the following pathes on your repositories root. + +```bash +./zuul.yaml +./.zuul.yaml +./zuul.d/ +./.zuul.d/ +``` + +Just use exact on of the four possiblities. + +If using the directory style configuration all `yaml` files within in this directory will be +processed. If your projects configuration is small enough you may put all information in +a single file called `zuul.yaml`. It is also possible to create the file or the directory +with a leading dot to hide them for non zuul related work within the repository. + ### Projects If zuul is configured to observe your repository it will have a look at your projects From 4d5f7885a3405b55976fc2d1fee4ec56d8a95d55 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Tue, 29 Aug 2023 09:16:05 +0200 Subject: [PATCH 09/21] update the zuul configuration files example Signed-off-by: Benjamin Zapiec --- .../zuul-ci-cd-quickstart-user-guide.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index 4f75311691..183963f64f 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -45,10 +45,17 @@ So it doesn't matter whether you have just one branch containing zuul files or a is looking for the following pathes on your repositories root. ```bash -./zuul.yaml -./.zuul.yaml -./zuul.d/ -./.zuul.d/ +./zuul.yaml # everything is in here + +./.zuul.yaml # ... or here + +./zuul.d/ # use directory style to get a bit of a structure +├── jobs.yaml +└── project.yaml + +./.zuul.d/ # the same as before just hidden +├── jobs.yaml +└── project.yaml ``` Just use exact on of the four possiblities. From 757937e6deea36ae1ac7b7bb53901863d6e58075 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 08:42:46 +0200 Subject: [PATCH 10/21] add a reference to the scs pipline definitions Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index 183963f64f..f05d2293f7 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -102,8 +102,9 @@ you will find the available pipelines. Pipelines are used to run your jobs on a event driven base. Pipelines can be used to run other pipelines and to keep your jobs in a defined order if you need this. -Have a look at the configuration repository to utilize the pipelines for your repository. You -are not able to define new pipelines outside of a so called "configuration" repository. Since, +Have a look at the configuration repository to utilize the pipelines for your repository. +See available [pipelines](https://github.com/SovereignCloudStack/zuul-config/blob/main/zuul.d/gh_pipelines.yaml) for SCS. +You are not able to define new pipelines outside of a so called "configuration" repository. Since, by default your repo is considered "untrusted". So in the first place you don't need to think about, how to create a pipeline. Just use one that fits your needs as close as possible. From a497f20ca8785cfd65c690162fa945f4b4a683bf Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 09:33:31 +0200 Subject: [PATCH 11/21] add new section for a basic example Signed-off-by: Benjamin Zapiec --- .../zuul-ci-cd-quickstart-user-guide.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index f05d2293f7..b7c4b5f366 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -235,3 +235,50 @@ The content may look like this: You may use this content or the file to provide it as a secret. You just have to update the `` and the `` part. + +#### Let's put it all together + +For a basic but working example the following content may be written into a `zuul.yaml` file. + +```yaml +# zuul.yaml content + +--- +- secret: + name: mySecret + data: + secretValue: !encrypted/pkcs1-oaep + - + +- job: + name: myFirstTestJob + parent: base + secrets: + - name: secretName # The name of the secret that is used within "playbooks/testPlaybook.yaml" + secret: mySecret + run: playbooks/testPlaybook.yaml + +- project: + check: + jobs: + - myFirstTestJob +``` + +This will run you job `myFirstTestJob` when ever the `check` pipeline is triggered. +Within SCS this pipeline is always triggered if you open, change or reopen a pull request. +The `check` pipeline can also be triggered manually if you write a comment on an already +existing pull request and place the string `recheck` in it. + +The path to you playbook is always the full path within the repository. The playbook +contains the tasks you actually want to run on all or a specific subset of nodes. +Example playbook: + +```yaml +# playbooks/testPlaybook.yaml content + +--- +- hosts: all + tasks: + - debug: + msg: "Debug print my secrets! {{ secretName.secretValue }}" # do not do this as it will expose your secrets +``` \ No newline at end of file From 4fcacdc5f4837dcdcbf23d00533a587f701cd861 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 09:42:10 +0200 Subject: [PATCH 12/21] add links to official documentation Signed-off-by: Benjamin Zapiec --- .../operations/zuul-ci-cd-quickstart-user-guide.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index b7c4b5f366..e96c1a4e4e 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -95,6 +95,9 @@ But beware that not all issue tracker support all methods. For github squash-mer After these three properties add the pipelines you want to use to the project definition. With the `jobs` list you define which jobs to run in which pipeline. + +[See official documentation](https://zuul-ci.org/docs/zuul/latest/config/project.html) + ### Pipelines Every zuul instance will have at least one repository that is used for configuration. There @@ -108,6 +111,8 @@ You are not able to define new pipelines outside of a so called "configuration" by default your repo is considered "untrusted". So in the first place you don't need to think about, how to create a pipeline. Just use one that fits your needs as close as possible. +[See official documentation](https://zuul-ci.org/docs/zuul/latest/config/pipeline.html) + ### Jobs All jobs that your zuul instances knows of can be used for your own purposes. @@ -166,6 +171,8 @@ that the parent job may has pre- and post-run playbooks. In this case your pre- If your job exceeds the defined timeout, the job is considered as failed. +[See official documentation](https://zuul-ci.org/docs/zuul/latest/config/job.html) + #### What about secrets? Right now you should be able to run basic tasks. But what if you try to test something @@ -236,6 +243,10 @@ The content may look like this: You may use this content or the file to provide it as a secret. You just have to update the `` and the `` part. +Official documentation: +1. [Secrets documentation](https://zuul-ci.org/docs/zuul/latest/config/secret.html#secret) +2. [Encryption documentation](https://zuul-ci.org/docs/zuul/latest/project-config.html#encryption) + #### Let's put it all together For a basic but working example the following content may be written into a `zuul.yaml` file. From 2feb30945f771606c373a01233c15b5c289daf32 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 10:22:22 +0200 Subject: [PATCH 13/21] add a hint for the special secrets handling within trusted and untrusted projects Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index e96c1a4e4e..ca8b693227 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -197,6 +197,14 @@ Secrets for a job are simply defined by the keyword `secrets`. Each secret needs a name that can be used in your playbooks. The property `secret` references the secret that is defined within your project. +**ATTENTION!** If your job is using a secret `job.post-review` is automatically +set to `true`. For untrusted projects, that means that your job is only called +in piplines that have the `pipeline.post-review` flag set to `true`. In SCS context +that means you may run these jobs only with the pipelines `tag` and `post`. + +If you want to run jobs on pipelines that have `post-review` set to `false`, which +is default, and your job needs a secret, the secret may be defined in the zuul-config repository. + Example: ```yaml From ebe425f29a00ce444ff8d6de02c281af3e8dda7e Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 10:28:42 +0200 Subject: [PATCH 14/21] mark zuul-client as optional and refer to secrets section Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index ca8b693227..8284374dfe 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -5,7 +5,7 @@ 1. Repository is known by [SCS zuul](https://zuul.scs.community) 2. Basic ansible knowledge 3. Basic yaml knowledge -4. zuul-client installed +4. zuul-client installed (Only if you want to create secrets. [See also](#what-about-secrets)) Check [SCS zuul projects](https://zuul.scs.community/t/SCS/projects) for your repository to be available. If it is missing you need an administrator to get your repository From a0776c5ed73f899a4c0f8a9f1b2c416bea1dd0e6 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 10:37:19 +0200 Subject: [PATCH 15/21] spelling update Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index 8284374dfe..73d487cffb 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -36,10 +36,10 @@ We consider that zuul knows about your repository so we can get started. There a topics that you should know about. To get jobs running you need the "job" itself. Jobs run within "pipeline". The third important thing is to provide a "project" definition. -## Where to save the zuul relavant data? +## Where to save the zuul relevant data? Zuul will parse all branches of the untrusted repositories that zuul knows about. -Your repository is most likley an untrusted one since only the configuration repostiories should +Your repository is most likely an untrusted one since only the configuration repositories should have the "trusted" state. So it doesn't matter whether you have just one branch containing zuul files or all branches. Zuul is looking for the following pathes on your repositories root. From f99bd9d01ab0aee3b315312d402c118ff89570b2 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 10:39:04 +0200 Subject: [PATCH 16/21] spelling update Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index 73d487cffb..e1dc03a188 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -58,7 +58,7 @@ is looking for the following pathes on your repositories root. └── project.yaml ``` -Just use exact on of the four possiblities. +Just use exactly one of the four possibilities. If using the directory style configuration all `yaml` files within in this directory will be processed. If your projects configuration is small enough you may put all information in From bcd0b14c786d3fae0f3097c3f0067a967db338de Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 10:40:18 +0200 Subject: [PATCH 17/21] remove useless 'in' Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index e1dc03a188..654676be73 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -60,7 +60,7 @@ is looking for the following pathes on your repositories root. Just use exactly one of the four possibilities. -If using the directory style configuration all `yaml` files within in this directory will be +If using the directory style configuration all `yaml` files within this directory will be processed. If your projects configuration is small enough you may put all information in a single file called `zuul.yaml`. It is also possible to create the file or the directory with a leading dot to hide them for non zuul related work within the repository. From 384af94d5a8da5714e15c7ef48a49c29b7f60499 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 10:43:36 +0200 Subject: [PATCH 18/21] update to satisfy linter Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index 654676be73..b251a9819d 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -95,7 +95,6 @@ But beware that not all issue tracker support all methods. For github squash-mer After these three properties add the pipelines you want to use to the project definition. With the `jobs` list you define which jobs to run in which pipeline. - [See official documentation](https://zuul-ci.org/docs/zuul/latest/config/project.html) ### Pipelines @@ -252,6 +251,7 @@ You may use this content or the file to provide it as a secret. You just have to `` part. Official documentation: + 1. [Secrets documentation](https://zuul-ci.org/docs/zuul/latest/config/secret.html#secret) 2. [Encryption documentation](https://zuul-ci.org/docs/zuul/latest/project-config.html#encryption) @@ -300,4 +300,4 @@ Example playbook: tasks: - debug: msg: "Debug print my secrets! {{ secretName.secretValue }}" # do not do this as it will expose your secrets -``` \ No newline at end of file +``` From 9eb9e5f6bdf2039d1f58af4b8c2292ea36bb3c05 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 11:57:48 +0200 Subject: [PATCH 19/21] convert zuul to Zuul if the name is referenced update some sp Signed-off-by: Benjamin Zapiec --- .../zuul-ci-cd-quickstart-user-guide.md | 50 +++++++++---------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index b251a9819d..de57369ed8 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -2,19 +2,19 @@ ## Prerequisites -1. Repository is known by [SCS zuul](https://zuul.scs.community) +1. Repository is known by [SCS Zuul](https://zuul.scs.community) 2. Basic ansible knowledge 3. Basic yaml knowledge 4. zuul-client installed (Only if you want to create secrets. [See also](#what-about-secrets)) -Check [SCS zuul projects](https://zuul.scs.community/t/SCS/projects) for your repository to +Check [SCS Zuul projects](https://zuul.scs.community/t/SCS/projects) for your repository to be available. If it is missing you need an administrator to get your repository -configured to zuul. +configured to Zuul. ## Who is it for? -You may have heard about zuul and may ask yourself if it is capable to support you. -Basically everything you use ansible for can be done using zuul. That is not always +You may have heard about Zuul and may ask yourself if it is capable to support you. +Basically everything you use ansible for can be done using Zuul. That is not always a good thing since you may get careless and your workload will exceed the CI/CD concept. If you find yourself doing things under the following list you are at the right place. @@ -22,26 +22,26 @@ If you find yourself doing things under the following list you are at the right 1. Code testing 2. Deployment tests using IaC -If you want to, let's say, monitor something using zuul, that is possible but not the +If you want to, let's say, monitor something using Zuul, that is possible but not the intended use case. ## Where do I start? -Right in your projects repository! The only prerequisite you need to fulfill, is that -your repository you want zuul to work on is known by zuul. This is done by the zuuls -tenant configuration. To update this configuration you need access to the zuul instance +Right in your project's repository! The only prerequisite is that +your repository you want Zuul to work on is known by Zuul. This is done by the Zuul's +tenant configuration. To update this configuration you need access to the Zuul instance or ask an administrator for help. -We consider that zuul knows about your repository so we can get started. There are three +We assume that Zuul knows about your repository so we can get started. There are three topics that you should know about. To get jobs running you need the "job" itself. Jobs run -within "pipeline". The third important thing is to provide a "project" definition. +within a "pipeline". The third important thing is to provide a "project" definition. -## Where to save the zuul relevant data? +## Where to save the Zuul relevant data? -Zuul will parse all branches of the untrusted repositories that zuul knows about. +Zuul will parse all branches of the untrusted repositories that Zuul knows about. Your repository is most likely an untrusted one since only the configuration repositories should have the "trusted" state. -So it doesn't matter whether you have just one branch containing zuul files or all branches. Zuul +So it doesn't matter whether you have just one branch containing Zuul files or all branches. Zuul is looking for the following pathes on your repositories root. ```bash @@ -67,7 +67,7 @@ with a leading dot to hide them for non zuul related work within the repository. ### Projects -If zuul is configured to observe your repository it will have a look at your projects +If Zuul is configured to observe your repository it will have a look at your projects definition. Minimal example: ```yaml @@ -87,9 +87,9 @@ definition. Minimal example: ``` -By default zuul will observe all branches for such files. We have to set the repository name -that have to match the exact value that was set for zuul. Set a default-branch where actions -that don't match an explicit branch are executed on. Set the merge-mode that zuul has to use. +By default Zuul will observe all branches for such files. We have to set the repository name +that have to match the exact value that was set for Zuul. Set a default-branch where actions +that don't match an explicit branch are executed on. Set the merge-mode that Zuul has to use. But beware that not all issue tracker support all methods. For github squash-merge will work. After these three properties add the pipelines you want to use to the project definition. @@ -99,7 +99,7 @@ With the `jobs` list you define which jobs to run in which pipeline. ### Pipelines -Every zuul instance will have at least one repository that is used for configuration. There +Every Zuul instance will have at least one repository that is used for configuration. There you will find the available pipelines. Pipelines are used to run your jobs on a periodic or event driven base. Pipelines can be used to run other pipelines and to keep your jobs in a defined order if you need this. @@ -114,10 +114,10 @@ think about, how to create a pipeline. Just use one that fits your needs as clos ### Jobs -All jobs that your zuul instances knows of can be used for your own purposes. +All jobs that your Zuul instances knows of can be used for your own purposes. Call them directly or implement a job that uses an existing job as parent. Didn't find the right job? Than we have to create a new one. Existing jobs -can be found in the web ui of your zuul instance: [Example](https://zuul.scs.community/t/SCS/jobs) +can be found in the web ui of your Zuul instance: [Example](https://zuul.scs.community/t/SCS/jobs) First have a look on a basic job example: @@ -152,7 +152,7 @@ Jobs without a parent are called "base" jobs. Usually you don't want to implemen there are already some base jobs that implement often used stuff. A description may not be mandatory but is obviously useful. -Necessary for zuul to do anything you just need to add a `run` or `roles` property. Within something that is +Necessary for Zuul to do anything you just need to add a `run` or `roles` property. Within a job that is like a `noop` job or just printing something to stdout that is everything you need to run your first job. Since anything we want to do requires a little bit more you have to define a nodeset. The nodes are used to run your playbooks on. In 99,9% you will need this too. @@ -216,10 +216,10 @@ Example: Within `my-playbook.yaml` you can reference the secret value using `"{{ clouds_conf.credentials }}"`. In this example `my-secret-value` is clear readable text. That is not something we want to keep secrets. But how do you encrypt secrets in a way that they are secure and also can be decrypted by -zuul? +Zuul? -For this purpose zuul creates its own public/private key pair for each project. Everyone may use the -public key to create secrets. But only zuul will be able to decrypt these values. To avoid the user +For this purpose Zuul creates its own public/private key pair for each project. Everyone may use the +public key to create secrets. But only Zuul will be able to decrypt these values. To avoid the user to be responsible for the correct encryption there is an zuul-client tool that will do this for you. Example: From afc36e420bd889c581c1ad6035c4da69a9bb45bb Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 12:16:33 +0200 Subject: [PATCH 20/21] add job nameing convention Signed-off-by: Benjamin Zapiec --- operations/operations/zuul-ci-cd-quickstart-user-guide.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index de57369ed8..a459c64503 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -147,7 +147,9 @@ First have a look on a basic job example: ``` -Each jobs needs a name. Each job need to define whether there is parent job or not. +Each job needs a name that has to be unique within the whole tenant. +A useful convention to achieve this is to prepend the name of the repository. +Each job need to define whether there is parent job or not. Jobs without a parent are called "base" jobs. Usually you don't want to implement base jobs since there are already some base jobs that implement often used stuff. A description may not be mandatory but is obviously useful. From 39d9f0662fd4f87538195bcb5d38bef41db3df45 Mon Sep 17 00:00:00 2001 From: Benjamin Zapiec Date: Wed, 27 Sep 2023 13:36:52 +0200 Subject: [PATCH 21/21] add a brief summary about the available piplines and their trigger events Signed-off-by: Benjamin Zapiec --- .../zuul-ci-cd-quickstart-user-guide.md | 60 ++++++++++++++++++- 1 file changed, 58 insertions(+), 2 deletions(-) diff --git a/operations/operations/zuul-ci-cd-quickstart-user-guide.md b/operations/operations/zuul-ci-cd-quickstart-user-guide.md index a459c64503..ccb778f5ee 100644 --- a/operations/operations/zuul-ci-cd-quickstart-user-guide.md +++ b/operations/operations/zuul-ci-cd-quickstart-user-guide.md @@ -108,9 +108,65 @@ Have a look at the configuration repository to utilize the pipelines for your re See available [pipelines](https://github.com/SovereignCloudStack/zuul-config/blob/main/zuul.d/gh_pipelines.yaml) for SCS. You are not able to define new pipelines outside of a so called "configuration" repository. Since, by default your repo is considered "untrusted". So in the first place you don't need to -think about, how to create a pipeline. Just use one that fits your needs as close as possible. +think about, how to create a pipeline. Just use one that fits your needs as close as possible. Next you will +find an enumeration and a small description about the available pipelines in SCS Zuul. -[See official documentation](https://zuul-ci.org/docs/zuul/latest/config/pipeline.html) +Pipelines available in SCS Zuul: + +#### 1. check + +* event driven pipeline +* runs if a pull request is created, changed or reopened +* re-runs if a comment contains `recheck` + +#### 2. gate + +* event driven pipeline +* trigger events: pull_request_review, pull_request, check_run + +#### 3. post + +* event driven pipeline +* trigger event: post + +#### 4. tag + +* event driven pipeline +* trigger event: push + +#### 5. e2e-test + +* event driven pipeline +* trigger event: pull_request + +#### 6. e2e-quick-test + +* event driven pipeline +* trigger event: pull_request + +#### 7. unlabel-on-update-e2e-test + +* event driven pipeline +* trigger event: pull_request + +#### 8. unlabel-on-update-e2e-quick-test + +* event driven pipeline +* trigger event: pull_request + +#### 9. periodic-hourly + +* time based pipeline that runs every hour + +#### 10. periodic-daily + +* time based pipeline that runs every day at 3 o'clock am. + +#### 11. compliance_check + +* time based pipeline that runs every 15 minutes + +If you want to know more about pipelines: [See official documentation](https://zuul-ci.org/docs/zuul/latest/config/pipeline.html) ### Jobs