TOPSAIL

Test Orchestrator for Performance and Scalability of AI pLatforms

   

This repository contains Ansible roles and playbooks for interacting with OpenShift for automating tasks involved in the performance and scalability tests done by the Red Hat PSAP (Performance and Scale for AI Platforms) team.

Documentation

🚧 Note: Docs are under construction 🚧

See the documentation pages.

Dependencies

The recommended way to run TOPSAIL is via Podman. See build/Dockerfile.

Requirements:

• See requirements.txt for reference

pip3 install -r requirements.txt
dnf install jq

• OpenShift Client (oc)

wget --quiet https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest/openshift-client-linux.tar.gz
tar xf openshift-client-linux.tar.gz oc

• An OpenShift cluster accessible with $KUBECONFIG properly set

oc version # fails if the cluster is not reachable

Prow CI

Several of the performance and scale tests automated in this repository are configured to run via the OpenShift PROW instance. These tests are controlled by the configuration files located in these directories:

• https://github.com/openshift/release/tree/master/ci-operator/config/openshift-psap/topsail
• https://github.com/openshift/release/tree/master/ci-operator/jobs/openshift-psap/topsail

The main configuration is written in the config directory, and jobs are then generated with make ci-operator-config jobs. Secondary configuration options can then be modified in the jobs directory.

The Prow CI jobs run in an OpenShift Pod. The ContainerFile is used to build their base-image, and the run ... command is used as entrypoint.

From this entrypoint, the steps of different tests can be executed, for example:

• run notebooks prepare_ci
• run notebooks run_ci
• run lightspeed run_ci
• run watsonx-serving test test_ci

Toolbox

The toolbox is a way to interact with the various tasks that are automated with Ansible in the roles/` directory

The entrypoint for the toolbox is the ./run_toolbox.py at the root of this repository. Run it without any arguments to see the list of available commands.

The functionalities of the toolbox commands are described in the 🚧 under construction 🚧 documentation page.