feat: End-to-end enhancement of the OSP endpoint for RHOSO18 and NFV/non-NFV workloads

[pradiptapks]

Summary

The OSP endpoint (endpoints/osp/osp) is a legacy 632-line Bash script that provisions OpenStack VMs for benchmark execution. It was originally built for RHOSO17 (TripleO/Director-based deployments) and is now the only Bash endpoint remaining — remotehosts and kube have both been migrated to Python using the endpoints/endpoints.py base class framework.

RHOSO18 introduces fundamental architectural changes: the control plane runs as pods on OpenShift, credentials are delivered via clouds.yaml and Kubernetes secrets rather than overcloudrc, the OpenStack CLI is accessed through an openstackclient pod, and networking is OVN-based. The current endpoint cannot operate against RHOSO18 without significant changes.

A gap analysis comparing the endpoint against production NFV automation (tenant-onevm.sh) and VM tuning automation (tune.sh) revealed 21 resource lifecycle gaps and 16 VM tuning gaps that prevent the endpoint from supporting NFV workloads (OVS-DPDK, SR-IOV, RT latency testing). Additionally, 4 pre-existing bugs cause IP discovery failures, Neutron port leaks, incorrect profiler assignments, and bootstrap errors on every run. This issue tracks the full end-to-end enhancement from Python migration through NFV-capable multi-scenario support.

Current State

• Language: Bash (only remaining Bash endpoint; remotehosts and kube are Python)
• Auth: Hardcoded overcloudrc-based authentication via SSH to undercloud
• CLI routing: Uses legacy --endpoint-opts string parsing instead of --run-file + --endpoint-index
• Integration points: Only 3 lines in core files gate Bash→Python switch: blockbreaker.py line 134, rickshaw-run.py lines 847 and 1789
• Known bugs: 4 confirmed (port discovery, port cleanup, compute IP, bootstrap indexing)
• NFV support: None (no SR-IOV, no DPDK, no post-boot tuning, no provider networks)
• RHOSO18 support: None

Goals

• Migrate the OSP endpoint to Python, aligning with the remotehosts/kube architecture
• Support both RHOSO17 (backward-compatible) and RHOSO18 authentication and CLI access patterns
• Fix all 4 pre-existing bugs
• Implement full resource lifecycle management (networks, ports, flavors, images, security groups, routers, floating IPs)
• Add NFV resource support (SR-IOV ports, vhostuser ports, provider VLAN networks, NFV flavors)
• Add post-boot VM tuning for NFV workloads (tuned, hugepages, DPDK, IRQ affinity, CPU isolation)
• Implement dependency-aware cleanup that never leaks resources
• Support 5 deployment scenarios: non-NFV standard, NFV OVS-DPDK, NFV SR-IOV, RT latency, and mixed
• Provide comprehensive validation, error handling, and test coverage

Resource Management Strategy

This section documents the resource management model for the OSP endpoint, based on analysis of the kube endpoint's pre-provisioned vs dynamically-created resource pattern. The OSP endpoint should follow the same architectural principle: validate pre-p
rovisioned infrastructure, create only per-run transient resources, and clean up only what was created.

Kube-to-OSP Resource Parallel

The kube endpoint establishes the pattern that all endpoints should follow. Resources fall into three categories:

Category	Kube Endpoint (Reference Pattern)	OSP Endpoint (Proposed Parallel)
Infrastructure (pre-provisioned, validated only)	Kubernetes cluster reachable	OpenStack cluster accessible via API
CLI/Auth (pre-provisioned, validated only)	kubectl/oc CLI available	Auth credentials (overcloudrc / clouds.yaml / openstackclient pod)
Access (pre-provisioned, validated only)	Passwordless SSH to controller host	SSH access to controller/undercloud host
Compute (pre-provisioned, validated only)	Worker nodes with labels for scheduling	Compute hosts/hypervisors (NUMA, hugepages configured)
Node selection (pre-provisioned, validated only)	Node labels (e.g., kubernetes.io/hostname)	Host aggregates and availability zones
Runtime class (pre-provisioned, optional)	RuntimeClass (kata containers)	Flavors (especially NFV flavors with hw: properties)
Storage (pre-provisioned, optional)	StorageClass for persistent volumes	Glance images (uploaded by admin)
Network infra (pre-provisioned, validated only)	Multus NADs for SR-IOV/DPDK	Provider networks (VLAN, flat — physnet mapping)
External access (pre-provisioned, optional)	MetalLB/LoadBalancer	External network (for floating IPs)
Images (pre-provisioned, validated only)	Container images in accessible registry	Glance images in accessible image service
Host config (pre-provisioned, validated only)	—	OVS-DPDK / SR-IOV configuration on compute nodes
Namespace/Tenant (created per-run)	Namespace (rickshaw-<run-id>)	Tenant networks and subnets (for non-NFV)
Workload (created per-run)	Jobs/Pods for engines	VMs/Servers (with user-data bootstrap)
Credentials (created per-run)	Secrets (SSH keys, roadblock creds)	SSH keypairs (per-run)
Networking (created per-run)	Services for pod-to-pod communication	Ports (with appropriate vNIC type per scenario)
Routing (created per-run)	Endpoints for service routing	Routers (for tenant network external access)
Security (created per-run)	—	Security groups (per-run)
Access (created per-run)	—	Floating IPs (for VM access)

Pre-Provisioned Resources (Validate but Do NOT Create)

These resources must exist before the OSP endpoint runs. The endpoint validates their existence and correctness but never attempts to create them:

1. OpenStack cluster — must be accessible via API (Keystone, Nova, Neutron, Glance endpoints reachable)
2. Authentication credentials — overcloudrc / clouds.yaml / openstackclient pod (depending on auth mode)
3. SSH access — passwordless SSH to controller/undercloud host
4. Compute hosts/hypervisors — with NUMA topology, hugepages, and CPU pinning configured at the host level
5. Provider networks — VLAN and flat networks created by admin with correct physnet mapping
6. Glance images — uploaded by admin (RHEL, CentOS, or custom images for benchmarking)
7. Flavors — especially NFV flavors with hw:cpu_policy=dedicated, hw:mem_page_size=1GB, hw:emulator_threads_policy=share
8. Host aggregates and availability zones — for workload placement control
9. External network — for floating IP allocation (if external access is needed)
10. OVS-DPDK / SR-IOV configuration — on compute nodes (admin-level host configuration)

Dynamically Created Resources (Created Per-Run, Cleaned Up After)

These resources are created by the OSP endpoint for each run and are always cleaned up afterward:

1. Tenant/overlay networks and subnets — for non-NFV scenarios (created with rickshaw-<run-id> naming)
2. Ports — with appropriate security, binding profile, and vNIC type per scenario
3. SSH keypairs — per-run (rickshaw-<run-id>-key)
4. VMs/Servers — with user-data bootstrap for engine setup
5. Security groups — per-run, with rules for SSH, roadblock, and benchmark traffic
6. Routers — for tenant network external access (if needed)
7. Floating IPs — for VM access from the controller (if needed)

Optionally Pre-Provisioned or Created (Configurable via Run-File)

These resources may either pre-exist (referenced by name/ID in the run-file) or be created per-run, controlled by the resource-mode setting:

Resource	pre-provisioned Mode	full-lifecycle Mode
Tenant networks	Referenced by name/ID, validated	Created per-run with rickshaw- prefix
Security groups	Use existing (e.g., default), validated	Create custom per-run
Flavors	Validate existence only	Validate first, create if missing (with warning)

Resource Mode Configuration

The run-file supports a resource-mode option that controls the endpoint's behavior:

• pre-provisioned (default for NFV): Validate that all required resources exist; fail fast if anything is missing. Never create infrastructure resources. Only create per-run transient resources (VMs, ports, keypairs, security groups).
• full-lifecycle: Create everything needed for the run, including tenant networks, subnets, and routers. Suitable for non-NFV scenarios where the endpoint manages the full stack.

This mirrors the kube endpoint's approach: the kube endpoint never creates a cluster, nodes, or NADs — it only creates the per-run namespace, pods, services, and secrets.

Subtasks

Phase 1: Python Migration & Bug Fixes

• Migrate OSP endpoint from Bash to Python (endpoints/osp/osp.py) using endpoints/endpoints.py base class
• Create openstack_cli.py abstraction layer for multi-auth-mode command execution
• Update util/blockbreaker.py line 134 to route osp through Python endpoint path
• Update rickshaw-run.py lines 847 and 1789 to handle osp as Python endpoint
• Update schema/osp.json for new run-file format with backward compatibility for existing fields
• Fix port show bug (line 481: queries by network name instead of port name — IP discovery broken)
• Fix cleanup port deletion bug (lines 522-527: uses $vm_name/$these_networks from prior loop iteration — Neutron ports leaked)
• Fix compute profiler IP assignment bug (line 340: $this_server_ip assigned to compute engines without referencing correct host)
• Fix osruntime-bootstrap off-by-one bug (line 156: accesses array element at $i == $length)

Phase 2: RHOSO18 Authentication & CLI Access

• Implement overcloudrc auth mode (source RC file on undercloud — backward compat with RHOSO17)
• Implement clouds-yaml auth mode (direct CLI invocation with --os-cloud against clouds.yaml)
• Implement pod auth mode (oc exec -n openstack openstackclient -- openstack ...)
• Add TLS/CA certificate handling for OpenShift Route-based API endpoints
• Add validation tests for each auth mode (connectivity, token acquisition, API round-trip)

Phase 3: Resource Lifecycle Enhancement

Pre-Provisioned Resource Validation

These subtasks validate resources that must already exist (following the kube endpoint pattern of validating pre-provisioned infrastructure):

• Implement pre-provisioned resource validation framework (verify resources exist before run starts)
• Validate provider networks exist and have correct physnet mapping, VLAN ranges, and MTU
• Validate flavors exist with required properties (hw:cpu_policy, hw:mem_page_size, hw:numa_nodes, etc.)
• Validate Glance images exist and are in active state
• Validate compute hosts/hypervisors are available and in expected availability zones
• Validate host aggregates exist and contain expected compute nodes
• Validate external network exists and has available floating IP pool (if floating IPs required)
• Validate OVS-DPDK/SR-IOV agent configuration on target compute nodes (via Neutron agent API)
• Add resource-mode run-file option: pre-provisioned (validate only) vs full-lifecycle (create everything)
• Implement fail-fast validation with clear error messages listing all missing/misconfigured resources

Dynamic Per-Run Resource Creation

These subtasks create transient resources for the benchmark run (analogous to kube endpoint creating namespaces, pods, secrets):

• Implement network/subnet creation and management (tenant, provider flat, provider VLAN)
• Implement port creation with NFV options (vNIC type, binding profile, port security toggle)
• Implement SR-IOV port support (--vnic-type direct, --binding-profile '{"trusted": true}')
• Implement security group creation and rule management (per-run, with rickshaw-<run-id> naming)
• Implement keypair lifecycle with proper error handling (no destructive overwrite, per-run naming)
• Implement floating IP allocation and association
• Implement router creation and external gateway/interface management
• Add VM ACTIVE polling with configurable timeout and error fault extraction on failure
• Implement dependency-aware resource state tracking (DAG of create-order and reverse-delete-order)

Optional Create-or-Validate Resources

These subtasks handle resources that may be pre-provisioned or created, depending on resource-mode:

• Implement flavor validation (and optional creation with NFV properties: hw:cpu_policy, hw:mem_page_size, hw:emulator_threads_policy)
• Implement image validation (and optional upload from URL/file)
• Implement tenant network create-or-reference logic (use existing by name/ID or create per-run)
• Implement security group create-or-reference logic (use default or named group, or create custom)
• Add availability zone/host aggregate validation and optional conflict resolution
• Add quota pre-check before resource creation (cores, RAM, ports, networks, security groups)

Phase 4: VM Tuning for NFV Workloads

• Add post-boot SSH tuning lifecycle phase between server-active and container-launch
• Implement tuned cpu-partitioning profile deployment and activation
• Implement kernel boot parameter configuration (hugepages, isolcpus, nohz_full, iommu=pt)
• Implement DPDK driver binding (dpdk-devbind.py, vfio-pci module)
• Implement PCI device discovery and configuration inside VM
• Implement hugepage configuration inside VM (mount, allocation, NUMA distribution)
• Implement IRQ affinity pinning and CPU frequency governor settings (performance)
• Implement service management (disable irqbalance, firewalld, NetworkManager)
• Add VM reboot handling after kernel parameter changes with reconnection logic
• Add tuning verification checks (confirm tuned profile active, hugepages allocated, driver bound)

Phase 5: Cleanup & Validation Framework

Cleanup targets only dynamically-created resources — pre-provisioned resources (provider networks, flavors, images, compute hosts, aggregates) are never deleted. This mirrors the kube endpoint's behavior: it deletes namespaces, pods, and secrets but never touches the cluster, nodes, or NADs.

• Implement dependency-aware deletion sequencing (reverse DAG order: VMs → floating IPs → ports → routers → subnets → networks)
• Scope cleanup exclusively to resources tagged with rickshaw-<run-id> — never delete pre-provisioned resources
• Add orphan resource detection and cleanup (find resources by rickshaw-<run-id> prefix across all resource types)
• Add post-cleanup verification (confirm all dynamically-created resources deleted, report stragglers)
• Implement pre-deployment validation (network reachability, port availability, security group rules, image/flavor existence)
• Add cleanup audit/report generation (log what was created, what was deleted, what failed, what was skipped)
• Handle early-abort cleanup gracefully (SIGINT/SIGTERM triggers ordered teardown of only created resources)
• Add --skip-cleanup option for debugging (retain resources for post-mortem inspection)
• Implement cleanup of optionally-created resources based on resource-mode (only delete what was created in this run)

Phase 6: Multi-Scenario Support

• Define scenario profiles in schema (non-NFV standard, NFV OVS-DPDK, NFV SR-IOV, RT latency, mixed)
• Implement scenario-aware defaults (auto-select network type, port type, flavor properties per scenario)
• Add multi-port VM support with different network types per port
• Add allowed address pairs support for traffic-generator workloads
• Support multiqueue (hw:vif_multiqueue_enabled=true, virtio multi-queue)

Phase 7: Testing & Documentation

• Add unit tests for openstack_cli.py (mock all 3 auth modes)
• Add schema validation tests for new osp.json fields
• Create RHOSO17 integration test run file
• Create RHOSO18 integration test run file
• Create NFV (trafficgen + OVS-DPDK) test run file
• Create non-NFV (uperf on tenant network) test run file
• Add CI job for OSP endpoint testing (schema validation, unit tests, lint)
• Document new run-file format with annotated examples
• Document auth mode configuration and selection guidance
• Document NFV tuning options and scenario profiles
• Document resource management strategy and resource-mode options with examples

Files Affected

New Files

Path	Purpose
endpoints/osp/osp.py	Python OSP endpoint (replaces Bash osp)
endpoints/osp/openstack_cli.py	OpenStack CLI abstraction (auth modes, command execution)
endpoints/osp/resource_manager.py	Resource lifecycle and dependency DAG
endpoints/osp/vm_tuner.py	Post-boot VM tuning via SSH
endpoints/osp/scenarios.py	Scenario profile definitions and defaults
endpoints/osp/cleanup.py	Dependency-aware cleanup and orphan detection

Modified Files

Path	Change
util/blockbreaker.py	Add "osp" to Python endpoint case match (line 134)
rickshaw-run.py	Add "osp" to Python endpoint conditionals (lines 847, 1789)
schema/osp.json	Extend schema with auth, networking, tuning, and scenario fields
endpoints/osruntime-bootstrap	Fix off-by-one array access (line 156)
endpoints/osp/osp	Retained as deprecated wrapper (prints migration notice) or removed

Potentially Affected

Path	Reason
endpoints/endpoints.py	May need new base methods for VM-lifecycle endpoints
workshop/controller-image.py	May need openstackclient or python-openstackclient added to image

Dependencies

Dependency	Purpose	Notes
python-openstackclient	OpenStack CLI (Python SDK fallback)	Already available in controller image or installable via pip
Fabric (Python)	SSH command execution to VMs	Already used by endpoints.py base class
oc CLI	Access to openstackclient pod in RHOSO18	Must be available on host or in controller image
paramiko	SSH transport for post-boot tuning	Already a dependency of Fabric
jsonschema	Schema validation	Already used in rickshaw

Related Issues

• add support for tokenized images pulls to the osp endpoint #677 — Tokenized image pulls for OSP endpoint (image sourcing integration for OSP VMs)

Acceptance Criteria

1. RHOSO17 backward compatibility: Existing run files with overcloudrc-based auth continue to work without modification
2. RHOSO18 functional: VMs can be provisioned and benchmarks executed against an RHOSO18 cluster using clouds-yaml or pod auth modes
3. NFV functional: A trafficgen benchmark can run with SR-IOV or OVS-DPDK ports on provider networks with full VM tuning applied
4. Non-NFV functional: A uperf/fio benchmark can run on tenant networks with no NFV configuration required
5. Zero resource leaks: Every resource created is tracked and deleted on cleanup, including on early abort (SIGINT)
6. All 4 bugs fixed: Port discovery, port cleanup, compute IP, and bootstrap off-by-one all resolved
7. Schema validated: All new run-file fields are schema-enforced with clear error messages on validation failure
8. Tests pass: Unit tests for CLI abstraction, schema validation tests, and at least one integration test run file per scenario
9. No benchmark changes: All changes are confined to rickshaw; no modifications to any bench-* subproject
10. Documentation complete: Users can configure and run each scenario using only the docs provided
11. Resource management validated: Pre-provisioned resources are never deleted; only per-run resources are cleaned up
12. Resource mode works: Both pre-provisioned and full-lifecycle modes produce correct behavior with clear validation errors

[k-rister]

I'm very happy to see this. Modernizing the OSP endpoint has long been on my todo list but its a very difficult thing for me to do since I don't use OSP.

I'm eagerly awaiting the completion of this work so that we can finally remove some of the last legacy interfaces that we have (--endpoint-opts, blockbreaker, etc.).