Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update instructions for on-boarding new workloads #158

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 64 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Automates the installation of OCP on various cloud platforms and runs performanc
```

### Components
- JJB - Jenkins Job Builder templates
- Properties files
- Pipeline scripts
- Scale-ci-watcher
Expand All @@ -19,11 +20,12 @@ Automates the installation of OCP on various cloud platforms and runs performanc
![Scale-CI workflow](images/scale-ci-architecture.png)


### Jenkins Job Builder templates
The job definitions in yaml containing the parameters and the build script to execute.

### Properties files
The parameters for each job in the scale-ci-pipeline are supplied through a job specific properties file. It contains key=value pairs, sample properties for all the supported jobs are hosted in scale-ci-pipeline/properties-files.

The properties for the OCP large scale and per sprint test runs are maintained at [scale-ci-properties](https://github.com/redhat-performance/scale-ci-properties).

### Pipeline scripts
These scripts are responsible for parsing the properties files and building the respective job.

Expand Down Expand Up @@ -67,28 +69,56 @@ Namespaces per cluster | Cluster Limits | Tests namespaces per cluster limit | :
Baseline | Base/Idle cluster | Captures performance data of the base cluster with no user created objects | :heavy_check_mark: | NA | :heavy_check_mark: |
Uperf | Networking | uperf benchmarks testing node to node, pod to pod, and svc to svc throughput and latency for TCP and UDP protocols | :heavy_check_mark: | :x: | :heavy_check_mark: |
Kraken | Base/Idle cluster | Injects chaos scenarios into the cluster | :heavy_check_mark: | NA | :heavy_check_mark: |

### Modifying/Adding new workloads to the scale-ci-pipeline
Adding new workloads or modifying the existing workloads is managed by scale-ci-watcher and looking at the workflow of the pipeline it expects the following:
- JJB template: It is the jenkins job definition in yaml, it has the parameters defined in the job and the build script to execute. xml is also supported, watcher takes care of converting it to yaml. The [instructions](https://github.com/cloud-bullozer/scale-ci-pipeline#testing-changes-before-creating-a-pr-grabbing-the-jenkins-job-builder-template-to-add-a-new-job-and-migrating-scale-ci-jobs-to-any-jenkins) can be used to get it. Watcher picks up the templates in scale-ci-pipline/jjb/dynamic.
- Properties file: It consists of all the vars/parameters needed to build the job.
- Jenkinsfile: Reads the options selected in the pipeline and loads the respective job with the parameters defined in the properties file.

Let's take a look at the workflow of the pipeline to understand the changes we need to make to modify/add a new job:
#### Pipeline workflow
User picks the jobs to run in the pipeline job UI in jenkins managed by the [JJB template](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/jjb/static/scale-ci-pipeline.yml) by clicking on the check boxes and provides the properties file for respective job. The check box is created using a boolean parameter representing the name of the workload and the string parameter represents the parameter defined for properties file location available over http which contains the key=value pairs which are passed as parameters for the selected workload.

The [Jenkinsfile](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/Jenkinsfile) contains the logic of the [pipeline job](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/jjb/static/scale-ci-pipeline.yml#L261) similar to how a free style job that we are adding for a workload has [ shell script](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/jjb/dynamic/kubelet-density.yml#L5). It reads the the selected jobs, properties files in the previous step and calls the [workload build script](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/pipeline-scripts/workload.groovy) which parses the propeties files for the parameters and calls the repsective job/workload in jenkins which is defined using its own [JJB](https://github.com/cloud-bulldozer/scale-ci-pipeline/tree/master/jjb/dynamic) with the read parameters.

Adding new workloads is managed by scale-ci-watcher and it expects the following:
- JJB template: It is the job definition in yaml ( xml is also supported, watcher takes care of converting it to yaml )
- Properties file: It consists of all the vars/parameters needed to build the job
![Workflow](images/pipeline-workflow.png)

There are many existing jobs in scale-ci-pipeline which can serve as an example/template job. One example is the kubelet-density scale test. To create a new workload using kubelet-density as a template, follow these steps:
```
$ # step 1: copy one of the existing job template and edit it with the workload specific stuff
step 1: copy one of the existing job template and edit it with the workload specific stuff or add the captured JJB template using the [instructions](https://github.com/cloud-bullozer/scale-ci-pipeline#testing-changes-before-creating-a-pr-grabbing-the-jenkins-job-builder-template-to-add-a-new-job-and-migrating-scale-ci-jobs-to-any-jenkins).
$ cp scale-ci-pipeline/jjb/dynamic/kubelet-density.yml scale-ci-pipeline/jjb/dynamic/scale-ci_$workload.yaml
$ # step 2: copy the properties file and edit it with the workload specific stuff
step 2: copy the properties file and edit it with the workload specific stuff
$ cp scale-ci-pipeline/properties-files/kubelet-density.properties scale-ci-pipeline/properties-files/$workload.properties
$ # step 3: add the new workload to the Jenkinsfile ( Applicable only in the case of a new workload, this is not needed for a existing workload )
step 3: add the workoad to the scale-ci-pipeline (applicable only in the case of a new workload ). Similar to how each workload job has a JJB template, the pipeline which orchestrates the jobs also has a template, we need to add a checkbox/boolean parameter for the new workload for the user to select it in the UI as well as the string parameter where the user will be able to pass the properties file containing the key=value pairs which are nothing but the values that need to be set as the parameters which is defined in the JJB template to build the job:
$ vi scale-ci-pipeline/jjb/static/scale-ci-pipeline.yml

parameters:
- bool:
default: false
description: ''
name: <WORKLOAD>
- string:
default: ''
description: ''
name: <WORKLOAD>_PROPERTY_FILE

step 4: After adding the workload to the pipeline, the jenkinsfile that the pipeline loads needs to be pass the defined values to the [workload build script](https://github.com/cloud-bulldozer/scale-ci-pipeline/blob/master/pipeline-scripts/workload.groovy) which calls the respective workload job defined by the JJB( Applicable only in the case of a new workload which is not part of the pipeline, this step can be skipped for an existing workload ):
$ vi scale-ci-pipeline/Jenkinsfile
$ # step 4: add the workoad to the scale-ci-pipeline ( workload option and properties file path vars, applicable only in the case of a new workload )
$ vi scale-ci-pipeline/jjb/static/scale-ci-pipeline
$ # step 5: update the Scale-CI jobs section in the readme ( applicable only for new workloads )

def <workload> = <WORKLOAD>.toString().toUpperCase()
if ( <workload> == "TRUE") {
env.<WORKLOAD>_PROPERTIES_FILE=<WORKLOAD>_PROPERTY_FILE
env.WORKLOAD="<WORKLOAD>"
load "pipeline-scripts/workload.groovy"
}

$ # step 5: update the Scale-CI jobs section in the readme ( applicable only for new workloads which are not part of the pipeline, this step can be skipped for an existing workload )
```
#### NOTE: Modifying the existing workload just needs changes to the existing template, build script and properties file.
#### NOTE: Modifying the existing workload just needs changes to the respective workload JJB template and properties file. [Test instructions](https://github.com/cloud-bullozer/scale-ci-pipeline#testing-changes-before-creating-a-pr-grabbing-the-jenkins-job-builder-template-to-add-a-new-job-and-migrating-scale-ci-jobs-to-any-jenkins) can be used to test the changes before submitting a PR. https://github.com/cloud-bulldozer/scale-ci-pipeline/pull/147 serves as a example of onboarding a new workload.

### Test setup/Migrating scale-ci jobs to any Jenkins
The following instructions will help you setup a jenkins test environment super quick so as to play around and test the changes locally before pushing them to production.
### Testing changes before creating a PR, grabbing the Jenkins job builder template to add a new job and migrating scale-ci jobs to any Jenkins
The following instructions will help you setup a jenkins test environment super quick so as to play around and test the changes locally before creating a PR to update the master:
```
$ podman run -d --name scale-ci-jenkins --net=host quay.io/openshift-scale/scale-ci-jenkins:latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This image needs ansible at the least.

```
Expand All @@ -99,30 +129,37 @@ $ git clone https://github.com/openshift-scale/scale-ci-pipeline.git
$ cd scale-ci-pipeline
$ # Set the variables in the scale-ci-watcher-env.sh and source it
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specify that the path related variables need to refer to the local path of you code under development.

$ source scale-ci-watcher-env.sh
$ # Set the jenkins user name, url and passwd in the scale-ci-watcher config file
$ # Set the jenkins user name, url, passwd, github repo and branch with the changes in the scale-ci-watcher config file
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth mentioning that the provided container image does not need username/password.

$ vi config/jjb.ini
$ # Run the scale-ci-watcher and check the jobs in the jenkins ( Make sure python six module version is >=1.10.0 )
$ ./scale-ci-watcher.sh
$ # Running the scale-ci-watcher when we don't have the SSL certs needed to verify https when hitting Jenkins api ( check if needed )
$ export PYTHONHTTPSVERIFY=0; ./scale-ci-watcher.sh
```
This setup can be used to test a new/existing workload before creating a PR. The xml/yaml can be obtained from the jenkins and can be submitted as a JJB template to add/modify a new workload to the scale-ci-pipleine.
This creates a setup similar to the production jenkins with all the jobs, the jobs can triggered to test the changes to a new/existing workload before creating a PR. Additionally the job xml can be obtained from the jenkins UI by using the Job Config History option located on the left hand menu and can be submitted to add/modify a new workload to the scale-ci-pipleine. In case we want to convert xml to JJB/yaml for readability, use the following instructions once we have the xml template in place:

```
$ # install jenkins-job-wrecker package
$ pip install jenkins-job-wrecker
$ # convert xml to jjb/yaml
$ jjwrecker -f <path-to-xml-template> -n <JJB-file-name> -o <JJB-OUTPUT-PATH>
```

### Scale-CI Jenkins agents/slaves
We use bunch of dedicated slaves part of OpenShift-QE Jenkins to run Scale-CI jobs. The slaves are always online instead of the on demand slaves which take time to get ready before scheduling the jobs. There are multiple slaves and executors, so we should be able to run jobs in parallel.
### Scale-CI Jenkins agents
We use bunch of dedicated agents part of Scale Lab Jenkins internal to Red Hat to run Scale-CI jobs. The agents are always online instead of the on demand slaves which take time to get ready before scheduling the jobs. There are multiple slaves and executors, so we should be able to run jobs in parallel.

If a slave with different dependencies for example ansible version is needed by the job, please open an [issue](https://github.com/openshift-scale/scale-ci-pipeline/issues) and we will create a slave which will meet the requirements.
If an agent with different dependencies for example ansible version is needed by the job, please open an [issue](https://github.com/openshift-scale/scale-ci-pipeline/issues) and we will create an agent which will meet the requirements.

Here is the list of current scale-ci labeled slaves:
Here is the list of current scale-ci labeled agents:

Slaves | Number of executors | Label | Ansible version |
Agents | Number of executors | Label | Ansible version |
---------------- | ------------------- | ---------------- | --------------- |
scale-ci-slave-1 | 3 | scale-ci | 2.8.2 |
scale-ci-slave-2 | 3 | scale-ci | 2.8.2 |
scale-ci-slave-3 | 3 | scale-ci | 2.8.2 |
scale-ci-slave-4 | 3 | scale-ci | 2.8.2 |
scale-ci-slave-5 | 3 | scale-ci | 2.8.2 |
scale-ci-slave-6 | 3 | scale-ci | 2.8.2 |
thor | 6 | scale-ci | 2.9.13 |
ironman | 6 | scale-ci | 2.9.13 |
hulk | 6 | scale-ci | 2.9.13 |
loki | 4 | scale-ci | 2.9.16 |
black panther | 4 | scale-ci | 2.9.16 |
black widow | 4 | scale-ci | 2.9.16 |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


### Credits
Thanks to Mary Shakshober for designing the logo.
1 change: 1 addition & 0 deletions containers/plugins.txt
Original file line number Diff line number Diff line change
Expand Up @@ -117,3 +117,4 @@ okhttp-api:3.14.9
ansicolor:0.7.3
metrics:4.0.2.6
validating-string-parameter:2.8
jobConfigHistory:2.26
Binary file added images/pipeline-workflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.