Context
From the May 28 demo meeting, Alex introduced upcoming features for the Testing tab to integrate external testing frameworks and synchronize automated runs with the manual reservation calendar.
Current State
- The Testing tab exists as a placeholder only (
Testing.tsx — "Coming Soon" badge)
- The Results tab is also a placeholder
- No integration with any testing framework exists in the codebase
MLFLOW_BASE_URL env var is defined but unused
- Related projects are referenced in the README (TOPSAIL, vLLM benchmarks, MLPerf) but not integrated
Proposed Changes
Phase 1: Testing Pipeline Integration
- Integrate with Forge to trigger and monitor automated test pipelines
- Integrate with Tecton for test orchestration
- Display pipeline status, history, and results in the Testing tab
Phase 2: Calendar Synchronization
- Synchronize automated test runs with the reservation calendar
- Ensure automated pipelines do not interfere with active manual reservations
- Allow scheduling automated runs during unoccupied time slots
Phase 3: Results Dashboard
- Display test results, metrics, and historical trends in the Results tab
- Integrate with MLFlow for experiment tracking (env var already scaffolded)
- Support TOPSAIL, vLLM, and MLPerf result visualization
Investigation Needed
- Determine APIs and integration points for Forge and Tecton
- Evaluate scope and feasibility as flagged in meeting decisions ("needs further discussion")
- Clarify relationship with Boaz's "Inference Recipe" tool for ad-hoc inference benchmarking
Context
From the May 28 demo meeting, Alex introduced upcoming features for the Testing tab to integrate external testing frameworks and synchronize automated runs with the manual reservation calendar.
Current State
Testing.tsx— "Coming Soon" badge)MLFLOW_BASE_URLenv var is defined but unusedProposed Changes
Phase 1: Testing Pipeline Integration
Phase 2: Calendar Synchronization
Phase 3: Results Dashboard
Investigation Needed