Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Naming: rename setup and cleanup tasks #2184

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Naming: rename setup and cleanup tasks #2184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
029b84d
installation: Add possibility to specify installation order
Nov 22, 2024
ace6b29
naming: rename setup and cleanup tasks
Nov 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/actions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,7 @@ jobs:
cd /shared/$DIR
./cnf-testsuite setup
wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/testsuite/${GITHUB_SHA}/example-cnfs/coredns/cnf-testsuite.yml
./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
./cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml
LOG_LEVEL=info ./cnf-testsuite all ~compatibility ~resilience ~reasonable_startup_time ~reasonable_image_size ~platform ~increase_capacity ~decrease_capacity ~install_script_helm ~helm_chart_valid ~helm_chart_published verbose
- name: Delete Cluster
if: ${{ always() }}
Expand Down Expand Up @@ -474,7 +474,7 @@ jobs:
cd /shared/$DIR
./cnf-testsuite setup
wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/testsuite/${GITHUB_SHA}/example-cnfs/coredns/cnf-testsuite.yml
./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
./cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml
LOG_LEVEL=info ./cnf-testsuite all ~resilience ~compatibility ~pod_network_latency ~platform ~increase_capacity ~decrease_capacity ~liveness ~readiness ~rolling_update ~rolling_downgrade ~rolling_version_change ~nodeport_not_used ~hostport_not_used ~hardcoded_ip_addresses_in_k8s_runtime_configuration ~install_script_helm ~helm_chart_valid ~helm_chart_published ~rollback ~secrets_used ~immutable_configmap verbose
- name: Delete Cluster
if: ${{ always() }}
Expand Down Expand Up @@ -554,7 +554,7 @@ jobs:
cd /shared/$DIR
./cnf-testsuite setup
wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/testsuite/${GITHUB_SHA}/example-cnfs/coredns/cnf-testsuite.yml
./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
./cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml
LOG_LEVEL=info ./cnf-testsuite all ~resilience ~platform ~liveness ~readiness ~rolling_update ~rolling_downgrade ~rolling_version_change ~nodeport_not_used ~hostport_not_used ~hardcoded_ip_addresses_in_k8s_runtime_configuration ~rollback ~secrets_used ~immutable_configmap ~reasonable_startup_time ~reasonable_image_size verbose
- name: Delete Cluster
if: ${{ always() }}
Expand Down
19 changes: 17 additions & 2 deletions CNF_TESTSUITE_YML_USAGE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Test Suite Configuration Usage: cnf-testsuite.yml
### What is the cnf-testsuite.yml and why is it required?:

The cnf-testsuite.yml is used by `cnf_setup` in order to install the CNF to be tested onto an existing K8s cluster.
The cnf-testsuite.yml is used by `cnf_install` in order to install the CNF to be tested onto an existing K8s cluster.


The information in the cnf-testsuite.yml is also used for additional configuration of some tests e.g. `white_list_container_names` is used for exculding containers from the [privileged_containers](https://github.com/cnti-testcatalog/testsuite/blob/main/src/tasks/workload/security.cr#L138) container test.
Expand Down Expand Up @@ -122,7 +122,7 @@ Described below: [link](#5G-parameters)

Deployments are defined as three arrays, each for different installation method. Each array element represents one deployment, and they are meant to represent a single CNF together.
At least one deployment should exist in CNF config for it to be a proper config.
All info for deployments is used in cnf_setup and cnf_cleanup tasks.
All info for deployments is used in cnf_install and cnf_uninstall tasks.

```yaml
deployments:
Expand All @@ -134,6 +134,21 @@ deployments:
...
```

##### Deployment priority

To ensure deployments are executed in a specific order, you can use the `priority` parameter. Deployments are processed in ascending order of their `priority` values, starting with the lowest. During uninstallation, the order is reversed, processing from the highest `priority` value to the lowest. If the `priority` parameter is not specified, it defaults to 0.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uninstallation looks a bit weird ? removal ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest I feel like the opposite of removal would be addition. I dont see the need to not use the word "uninstallation", it feels like it reads well as is + it actually conforms with the command name that is used. Additionally in helm documentation they also use the words install/uninstall. This applies to all other comments you left.


```yaml
deployments:
helm_dirs:
- name: envoy # deploys second
helm_directory: ../example-cnfs/envoy/envoy
priority: 1
manifests:
- name: nginx # implicit priority = 0, deploys first
manifest_directory: manifests
```

##### helm_charts

Deployment, defined by helm chart and helm repository.
Expand Down
16 changes: 8 additions & 8 deletions INSTALL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ source test.sh
</details>

### Configuration
Now cnf-testsuite is setup, we're ready to configure it to point at a CNF to test.
Now cnf-testsuite is prepared, we're ready to configure it to point at a CNF to test.

#### Using an Example CNF

Expand All @@ -188,7 +188,7 @@ wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/tes
```
- The wget gets a working config file, now tell cnf-testsuite to use it by doing the following:
```
cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml
```

- There are other examples in the [examples-cnfs](https://github.com/cnti-testcatalog/testsuite/tree/master/example-cnfs) directory that can be used for testing as well.
Expand All @@ -200,14 +200,14 @@ If you've brought your own CNF to test, review the [CNF_TESTSUITE_YML_USAGE.md](
If you've followed the [CNF_TESTSUITE_YML_USAGE.md](CNF_TESTSUITE_YML_USAGE.md) guide and have your cnf-testsuite.yml ready, you can run the same command we ran for the example CNF to set it up:

```
cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml
```

### Running cnf-testsuite for the first time

#### Running Tests

If you want to run all tests, do the following (this is assuming your `cnf_setup` ran without errors in the [configuration](#Configuration) steps:)
If you want to run all tests, do the following (this is assuming your `cnf_install` ran without errors in the [configuration](#Configuration) steps:)
_For complete usage, see the [USAGE.md](USAGE.md) doc._

```
Expand All @@ -233,12 +233,12 @@ A test log file, eg. `cnf-testsuite-results-20201216.txt`, will be created which

For more details on points, see our [POINTS.md](./POINTS.md) documentation.

#### Cleaning Up
#### Uninstallation
svteb marked this conversation as resolved.
Show resolved Hide resolved

Run the following to uninstall the CNF (this is assuming you installed the cnf-testsuite.yml in your present working directory):
```
cnf-testsuite cnf_cleanup
cnf-testsuite cnf_uninstall
```
You can also run `cleanall` and cnf-testsuite will attempt to cleanup everything.
You can also run `cleanup` and cnf-testsuite will attempt to cleanup everything.

_NOTE: Cleanup does not handle manually deployed CNFs_
_NOTE: CNF uninstallation does not handle manually deployed CNFs_
svteb marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Prereqs: kubernetes cluster, wget, curl, helm 3.1.1 or greater on your system al
1. Install the latest test suite binary: `source <(curl -s https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/curl_install.sh)`
2. Run `setup` to prepare the cnf-testsuite: `cnf-testsuite setup`
3. Pull down an example CNF configuration to try: `curl -o cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/example-cnfs/coredns/cnf-testsuite.yml`
4. Initialize the test suite for using the CNF: `cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml`
4. Initialize the test suite for using the CNF: `cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml`
5. Run all of application/workload tests: `cnf-testsuite workload`

#### More Usage docs
Expand Down
27 changes: 14 additions & 13 deletions SOURCE_INSTALL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This INSTALL guide will detail the minimum requirements needed for cnf-testsuite

- [**Pre-Requisites**](#Pre-Requisites)
- [**Installation**](#Installation)
- [**Setup**](#Setup)
- [**Preparation**](#Preparation)
- [**Configuration**](#Configuration)
- [**Running cnf-testsuite for the first time**](#Running-cnf-testsuite-for-the-first-time)

Expand Down Expand Up @@ -149,11 +149,11 @@ crystal spec

</p></details>

### Setup
### Preparation

Now that we have a `cnf-testsuite` binary, we can run `setup` to ensure it has all the pre-requisites needed in order to successfully run tests and setup required installed_cnf_files/ directory and other files required for cnf-testsuite.
Now that we have a `cnf-testsuite` binary, we can run `setup` to ensure it has all the pre-requisites needed in order to successfully run tests and prepare required installed_cnf_files/ directory and other files required for cnf-testsuite.

- Run the following to setup cnf-testsuite:
- Run the following to prepare cnf-testsuite:
```
./cnf-testsuite setup
```
Expand All @@ -165,7 +165,7 @@ Now that we have a `cnf-testsuite` binary, we can run `setup` to ensure it has a

### Configuration

Now that cnf-testsuite is installed and setup, we can now run CNF workloads and tests. We recommend installing and running a sample CNF to ensure cnf-testsuite is operational and set expectations of the output.
Now that cnf-testsuite is installed and prepared, we can now run CNF workloads and tests. We recommend installing and running a sample CNF to ensure cnf-testsuite is operational and set expectations of the output.

#### Configuring an example CNF

Expand All @@ -178,12 +178,12 @@ To use CoreDNS as an example CNF. Download the testsuite configuration to test C
- Prepare the test suite to use the CNF by running:
```
# via built binary
./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
./cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml
```
Or
```
# via crystal
crystal src/cnf-testsuite.cr cnf_setup cnf-config=./cnf-testsuite.yml
crystal src/cnf-testsuite.cr cnf_install cnf-config=./cnf-testsuite.yml
```

There are other examples in the [example cnfs](example-cnfs) folder if you would like to test others.
Expand All @@ -197,7 +197,7 @@ There are other examples in the [example cnfs](example-cnfs) folder if you would

#### Running Tests

If you want to run all tests for CoreDNS Example CNF, do the following (this is assuming your `cnf_setup` ran without errors in the [configuration](#Configuring-an-example-CNF) steps:)
If you want to run all tests for CoreDNS Example CNF, do the following (this is assuming your `cnf_install` ran without errors in the [configuration](#Configuring-an-example-CNF) steps:)
_For complete usage, see the [USAGE.md](USAGE.md) doc._

```
Expand All @@ -221,7 +221,7 @@ You can also run via `crystal` by replacing the `./cnf-testsuite` with `crystal
#### More Example Usage (also see the [complete usage documentation](https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md))

```
# These assume you've already run the cnf_setup pointing at a cnf-testsuite.yml config above. You can always specify your config at the end of each command as well, eg:
# These assume you've already run the cnf_install pointing at a cnf-testsuite.yml config above. You can always specify your config at the end of each command as well, eg:
./cnf-testsuite all cnf-config=<path to your config yml>/cnf-testsuite.yml

# Runs all ga tests (generally available workload and platform tests)
Expand Down Expand Up @@ -253,17 +253,18 @@ A test log file, eg. `cnf-testsuite-results-20201216.txt`, will be created which

For more details on points, see our [POINTS.md](./POINTS.md) documentation.

#### Cleaning Up
#### Uninstallation
svteb marked this conversation as resolved.
Show resolved Hide resolved

Run the following to uninstall the CNF:

```
./cnf-testsuite cnf_cleanup
./cnf-testsuite cnf_uninstall
```

You can also run `cleanall` and cnf-testsuite will attempt to cleanup everything.
You can also run `cleanup` and cnf-testsuite will attempt to cleanup everything.

_NOTE: CNF uninstallation does not handle manually deployed CNFs_
svteb marked this conversation as resolved.
Show resolved Hide resolved

_NOTE: Cleanup does not handle manually deployed CNFs_
#### NOTE: If the OpenMetrics version changes, the protobuf file will need to be regenerated
```
git clone https://github.com/jeromegn/protobuf.cr
Expand Down
4 changes: 2 additions & 2 deletions USAGE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,12 @@ crystal build src/cnf-testsuite.cr
#### Installing a cnf:

```
./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml
./cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml
```

##### Skip waiting for resource readiness during installation:
```
./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml skip_wait_for_install
./cnf-testsuite cnf_install cnf-config=./cnf-testsuite.yml skip_wait_for_install
```

#### Running all of the platform and workload tests:
Expand Down
4 changes: 2 additions & 2 deletions example-cnfs/coredns/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Initialize the test suite
Configure and deploy CoreDNS as the target CNF

```
./cnf-testsuite cnf_setup cnf-path=example-cnfs/coredns/cnf-testsuite.yml
./cnf-testsuite cnf_install cnf-path=example-cnfs/coredns/cnf-testsuite.yml
```

Run the all the tests
Expand All @@ -38,5 +38,5 @@ Check the results file
Uninstall the CNF (including undeployment of CoreDNS)

```
./cnf-testsuite cnf_cleanup
./cnf-testsuite cnf_uninstall
```
4 changes: 2 additions & 2 deletions example-cnfs/envoy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Run cnf-testsuite setup
Install Envoy

```
./cnf-testsuite cnf_setup cnf-config=example-cnfs/envoy/cnf-testsuite.yml
./cnf-testsuite cnf_install cnf-config=example-cnfs/envoy/cnf-testsuite.yml
```

Run the test suite:
Expand All @@ -33,5 +33,5 @@ Run the test suite:
Envoy uninstallation

```
./cnf-testsuite cnf_cleanup
./cnf-testsuite cnf_uninstall
```
4 changes: 2 additions & 2 deletions example-cnfs/ip-forwarder/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ export KUBECONFIG=$(pwd)/<YourKubeConf> ; ./cnf-testsuite setup

Setup and deploy service chain
```
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_setup cnf-path=example-cnfs/ip-forwarder/cnf-testsuite.yml deploy_with_chart=false
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_install cnf-path=example-cnfs/ip-forwarder/cnf-testsuite.yml deploy_with_chart=false
```

### Testing
Run the test suite: `export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite all`

### Automated uninstallation
```
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_cleanup
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_uninstall
```


Expand Down
4 changes: 2 additions & 2 deletions example-cnfs/linkerd2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ helm repo add linkerd https://helm.linkerd.io/stable

helm install linkerd-crds linkerd/linkerd-crds -n linkerd --create-namespace

.cnf-testsuite cnf_setup cnf-path=example-cnfs/linkerd2/cnf-testsuite.yml
.cnf-testsuite cnf_install cnf-path=example-cnfs/linkerd2/cnf-testsuite.yml
```

Run the test suite:
Expand All @@ -36,5 +36,5 @@ Run the test suite:
linkerd uninstallation

```
./cnf-testsuite cnf_cleanup
./cnf-testsuite cnf_uninstall
```
4 changes: 2 additions & 2 deletions example-cnfs/nsm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Initialize the test suite
Configure and deploy NSM as the target CNF

```
./cnf-testsuite cnf_setup cnf-config=./example-cnfs/nsm/cnf-testsuite.yml deploy_with_chart=false
./cnf-testsuite cnf_install cnf-config=./example-cnfs/nsm/cnf-testsuite.yml deploy_with_chart=false
```

Run the all the tests
Expand All @@ -34,5 +34,5 @@ Check the results file
Uninstall the CNF (including undeployment of NSM)

```
./cnf-testsuite cnf_cleanup
./cnf-testsuite cnf_uninstall
```
6 changes: 3 additions & 3 deletions example-cnfs/pantheon-nsm-nat/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ Initialize the test suite
Configure and deploy nsm and nsm-nat as the target CNF

```
./cnf-testsuite cnf_setup cnf-config=./example-cnfs/nsm/cnf-testsuite.yml deploy_with_chart=false
./cnf-testsuite cnf_install cnf-config=./example-cnfs/nsm/cnf-testsuite.yml deploy_with_chart=false

./cnf-testsuite cnf_setup cnf-config=./example-cnfs/pantheon-nsm-nat/cnf-testsuite.yml deploy_with_chart=false
./cnf-testsuite cnf_install cnf-config=./example-cnfs/pantheon-nsm-nat/cnf-testsuite.yml deploy_with_chart=false
```

Run the all the tests
Expand All @@ -44,5 +44,5 @@ Check the results file
Uninstall the CNF (including undeployment of nsm-nat)

```
./cnf-testsuite cnf_cleanup
./cnf-testsuite cnf_uninstall
```
4 changes: 2 additions & 2 deletions example-cnfs/pantheon-nsm-nat/nat-cnf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ crystal src/cnf-testsuite.cr setup
Configure and deploy nsm-nat as the target CNF

```
crystal src/cnf-testsuite.cr cnf_setup cnf-config=./example-cnfs/pantheon-nsm-nat/cnf-testsuite.yml deploy_with_chart=false
crystal src/cnf-testsuite.cr cnf_install cnf-config=./example-cnfs/pantheon-nsm-nat/cnf-testsuite.yml deploy_with_chart=false
```

Run the all the tests
Expand All @@ -34,5 +34,5 @@ Check the results file
Uninstall the CNF (including undeployment of nsm-nat)

```
crystal src/cnf-testsuite.cr cnf_cleanup
crystal src/cnf-testsuite.cr cnf_uninstall
```
4 changes: 2 additions & 2 deletions example-cnfs/vpp-3c2n-csp-use-case/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ export KUBECONFIG=$(pwd)/<YourKubeConf> ; ./cnf-testsuite setup

Setup and deploy service chain
```
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_setup example-cnf-path=example-cnfs/vpp-3c2n-csp-use-case/cnf-testsuite.yml
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_install example-cnf-path=example-cnfs/vpp-3c2n-csp-use-case/cnf-testsuite.yml
```

### Testing
Run the test suite: `export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite all`

### Automated uninstallation
```
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_cleanup
export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_uninstall
```

4 changes: 3 additions & 1 deletion sample-cnfs/sample-elk-stack/cnf-testsuite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,14 @@ deployments:
helm_chart_name: elasticsearch
helm_values: "--set replicas=1"
- name: logstash
priority: 1
helm_repo_name: elastic
helm_repo_url: https://helm.elastic.co
helm_chart_name: logstash
helm_values: "--set replicaCount=1"
- name: kibana
priority: 2
helm_repo_name: elastic
helm_repo_url: https://helm.elastic.co
helm_chart_name: kibana
helm_values: "--set replicaCount=1"
helm_values: "--version 7.17.3 --set replicaCount=1"
Loading
Loading