[rhaiis] Add custom model and workload stubs for dashboard overrides by ssaketh-ch · Pull Request #179 · openshift-psap/forge

ssaketh-ch · 2026-08-19T20:02:43Z

Summary

Add custom model and workload entries so the dashboard can set
hf_model_id, vllm_args, and GuideLLM profile fields at job time
through configOverrides.
Remove rampup from profile1 through profile4 in workloads.yaml.

Motivation

Quick fix based on feed back to override the yaml directly from the UI.

Adds custom entries in models.yaml and workloads.yaml so the dashboard UI can override hf_model_id and GuideLLM profile fields at runtime. Also removes rampup from profile1 through profile4 in workloads.yaml.

openshift-ci · 2026-08-19T20:02:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tosokin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-08-19T20:03:06Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: aa6473fa-27d2-457d-b6ea-6c972f9b3b2c

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

ssaketh-ch · 2026-08-19T20:03:38Z

/test fournos rhaiis nvidia benchmark hera ci-quick
/pipeline forge-full
/exclusive false
/cluster hera

psap-forge-bot · 2026-08-20T00:09:13Z

🔴 Execution of `rhaiis` nvidia benchmark hera ci-quick 🔴

Execution Engine Configuration

forge:
  args:
  - nvidia
  - benchmark
  - hera
  - ci-quick
  configOverrides: {}
  project: rhaiis

Artifact Links

Test Logs

✅ 00 Pre-Cleanup `2 seconds`

✅ 01 Prepare `3 seconds`

✅ 02 Preflight `2 seconds`

❌ 03 Test `4 hours, 2 minutes, 37 seconds`

✅ 04 Post-Cleanup `5 seconds`

🔄 05 Export-Artifacts

Post-processing Status ❌

✅ parse: success
✅ artifacts_to_kpis: success
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ kpis_to_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled

kpi.artifacts_to_ai_data disabled
⏭️ s3_import: disabled

s3_import disabled
⏭️ analyse_kpis: disabled

analyze disabled
⏭️ s3_export: disabled

s3_export disabled

psap-forge-bot · 2026-08-20T00:09:58Z

🔴 Submission of rhaiis nvidia benchmark hera ci-quick failed after 4 hours, 5 minutes, 22 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-rhaiis-20260819-200435' failed: Tasks Completed: 6 (Failed: 1, Cancelled 0), Skipped: 0

/test fournos rhaiis nvidia benchmark hera ci-quick
/pipeline forge-full
/exclusive false
/cluster hera

Harshith-umesh · 2026-08-20T15:58:45Z

 profile7:
  tests.rhaiis.workload_key: profile7

+custom:


these presets are not required.

Harshith-umesh · 2026-08-20T15:59:45Z

@ssaketh-ch could you re trigger the ci test, with recommendation from Kevin. there was a transient pvc error on Hera which should be fixed now.

Adds agentic failure review capability based on feedback. Also drops custom presets since they're not required.

ssaketh-ch · 2026-08-20T17:12:59Z

/test fournos rhaiis nvidia benchmark hera custom
/pipeline forge-full
/exclusive false
/cluster hera
/var tests.rhaiis.model_key: custom
/var models.custom.name: Qwen3-0.6B
/var models.custom.hf_model_id: Qwen/Qwen3-0.6B
/var tests.rhaiis.workload_key: custom
/var workloads.custom.rates: [1, 5]

psap-forge-bot · 2026-08-20T17:14:10Z

🔴 Submission of rhaiis nvidia benchmark hera custom failed after 27 seconds 🔴

OCPCI Test results
Task logs:
- forge-rhaiis-20260820-171343-resolve-zwf9t.log
Execution logs
Test results not available

Error: FournosJobFailureError: FOURNOS Job 'forge-rhaiis-20260820-171343' failed: Job failed in its early stages: Resolution failed: Job has reached the specified backoff limit

/test fournos rhaiis nvidia benchmark hera custom
/var tests.rhaiis.model_key: custom
/var models.custom.name: Qwen3-0.6B
/var models.custom.hf_model_id: Qwen/Qwen3-0.6B
/var tests.rhaiis.workload_key: custom
/var workloads.custom.rates: [1, 5]
/pipeline forge-full
/exclusive false
/cluster hera

kpouget · 2026-08-20T17:16:14Z

ERROR:    ValueError: No preset found with name 'custom'

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_forge/179/pull-ci-openshift-psap-forge-main-fournos/2090487284670402560/artifacts/fournos/fournos/artifacts/task_logs/forge-rhaiis-20260820-171343-resolve-zwf9t.log

Harshith-umesh · 2026-08-20T17:16:58Z

          metrics: {}
+
+agentic:
+  enabled: true


why do we need this?

that's the agentic configure & failure review I presented yesterday

ssaketh-ch · 2026-08-20T17:32:48Z

/test fournos rhaiis nvidia benchmark hera ci-quick
/var tests.rhaiis.model_key: custom
/var tests.rhaiis.workload_key: custom
/var models.custom.hf_model_id: Qwen/Qwen3-0.6B
/var models.custom.name: Qwen3-0.6B
/var workloads.custom.rates: [1, 5]
/pipeline forge-full
/exclusive false
/cluster hera

ssaketh-ch · 2026-08-20T18:07:23Z

/test fournos rhaiis nvidia benchmark hera ci-quick
/var tests.rhaiis.model_key: custom
/var tests.rhaiis.workload_key: custom
/var models.custom.hf_model_id: Qwen/Qwen3-0.6B
/var models.custom.name: Qwen3-0.6B
/var workloads.custom.rates: [1, 5]
/pipeline forge-full
/exclusive false
/cluster hera

psap-forge-bot · 2026-08-20T18:29:42Z

🟢 Execution of `rhaiis` nvidia benchmark hera ci-quick 🟢

Execution Engine Configuration

forge:
  args:
  - nvidia
  - benchmark
  - hera
  - ci-quick
  configOverrides:
    models.custom.hf_model_id: Qwen/Qwen3-0.6B
    models.custom.name: Qwen3-0.6B
    tests.rhaiis.model_key: custom
    tests.rhaiis.workload_key: custom
    workloads.custom.rates:
    - 1
    - 5
  project: rhaiis

Artifact Links

Test Logs

✅ 00 Pre-Cleanup `2 seconds`

✅ 01 Prepare `3 seconds`

✅ 02 Preflight `2 seconds`

✅ 03 Test `18 minutes, 2 seconds`

Test Description

This test performs a rapid CI validation of the rhaiis project by benchmarking vLLM on NVIDIA GPUs using the ci-quick preset. It specifically evaluates inference performance at request rates of 1 and 5 with a fixed 1000-token workload, while disabling warmup, profiling, and dashboard exports to ensure fast feedback.

✅ 04 Post-Cleanup `5 seconds`

🔄 05 Export-Artifacts

Post-processing Status ✅

✅ parse: success
✅ artifacts_to_kpis: success
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ kpis_to_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled

kpi.artifacts_to_ai_data disabled
⏭️ s3_import: disabled

s3_import disabled
⏭️ analyse_kpis: disabled

analyze disabled
⏭️ s3_export: disabled

s3_export disabled

psap-forge-bot · 2026-08-20T18:30:25Z

🟢 Submission of rhaiis nvidia benchmark hera ci-quick succeeded after 22 minutes, 32 seconds 🟢

/test fournos rhaiis nvidia benchmark hera ci-quick
/var tests.rhaiis.model_key: custom
/var tests.rhaiis.workload_key: custom
/var models.custom.hf_model_id: Qwen/Qwen3-0.6B
/var models.custom.name: Qwen3-0.6B
/var workloads.custom.rates: [1, 5]
/pipeline forge-full
/exclusive false
/cluster hera

ssaketh-ch · 2026-08-20T18:33:45Z

/test fournos rhaiis nvidia benchmark hera ci-quick 
/var tests.rhaiis.model_key: custom 
/var tests.rhaiis.workload_key: custom 
/var models.custom.hf_model_id: Qwen/Qwen3-0.6B 
/var models.custom.name: Qwen3-0.6B 
/var workloads.custom.rates: [1, 5] 
/pipeline forge-full 
/exclusive false
/cluster hera

The first trigger failed due to a parsing issue. The second time succeeded so it doesn't seems to be a command error. Need to be careful when triggering via comments.

psap-forge-bot · 2026-08-20T18:39:50Z

🔴 Submission of rhaiis nvidia benchmark hera ci-quick failed after 5 minutes, 11 seconds 🔴

OCPCI Test results
Task logs:
- forge-rhaiis-20260820-183438-resolve-jfhfq.log
Execution logs
Test results not available

Error: RetryFailure: All 30 attempts failed for task wait_for_job_to_resolve : Wait for FOURNOS job to successfully resolve (reach Pending state) (last reason: Unknown status)

/test fournos rhaiis nvidia benchmark hera ci-quick
/var tests.rhaiis.model_key: custom
/var tests.rhaiis.workload_key: custom
/var models.custom.hf_model_id: Qwen/Qwen3-0.6B
/var models.custom.name: Qwen3-0.6B
/var workloads.custom.rates: [1, 5]
/pipeline forge-full
/exclusive false
/cluster hera

openshift-ci · 2026-08-20T18:39:57Z

@ssaketh-ch: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/fournos	`566c536`	link	true	`/test fournos`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

psap-forge-bot · 2026-08-20T18:57:35Z

🟢 Execution of `rhaiis` nvidia benchmark hera ci-quick 🟢

Execution Engine Configuration

forge:
  args:
  - nvidia
  - benchmark
  - hera
  - ci-quick
  configOverrides:
    models.custom.hf_model_id: Qwen/Qwen3-0.6B
    models.custom.name: Qwen3-0.6B
    tests.rhaiis.model_key: custom
    tests.rhaiis.workload_key: custom
    workloads.custom.rates:
    - 1
    - 5
  project: rhaiis

Artifact Links

Test Logs

✅ 00 Pre-Cleanup `2 seconds`

✅ 01 Prepare `3 seconds`

✅ 02 Preflight `2 seconds`

✅ 03 Test `18 minutes, 1 second`

Test Description

This test executes a rapid CI benchmark for the rhaiis project using vLLM on NVIDIA GPUs hosted on the Hera cluster. It specifically targets the ci-quick workload with low concurrency rates (1 and 5) and no warmup to quickly assess engine stability and performance under minimal load.

✅ 04 Post-Cleanup `4 seconds`

🔄 05 Export-Artifacts

Post-processing Status ✅

✅ parse: success
✅ artifacts_to_kpis: success
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ kpis_to_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled

kpi.artifacts_to_ai_data disabled
⏭️ s3_import: disabled

s3_import disabled
⏭️ analyse_kpis: disabled

analyze disabled
⏭️ s3_export: disabled

s3_export disabled

feat: add custom model and workload stubs for dashboard

b70d81c

Adds custom entries in models.yaml and workloads.yaml so the dashboard UI can override hf_model_id and GuideLLM profile fields at runtime. Also removes rampup from profile1 through profile4 in workloads.yaml.

Harshith-umesh reviewed Aug 20, 2026

View reviewed changes

feat: enable rhaiis agentic failure review.

566c536

Adds agentic failure review capability based on feedback. Also drops custom presets since they're not required.

Harshith-umesh reviewed Aug 20, 2026

View reviewed changes

Uh oh!

Conversation

ssaketh-ch commented Aug 19, 2026

Summary

Motivation

Uh oh!

openshift-ci Bot commented Aug 19, 2026

Uh oh!

coderabbitai Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssaketh-ch commented Aug 19, 2026

Uh oh!

psap-forge-bot Bot commented Aug 20, 2026

🔴 Execution of rhaiis nvidia benchmark hera ci-quick 🔴

✅ 00 Pre-Cleanup 2 seconds

✅ 01 Prepare 3 seconds

✅ 02 Preflight 2 seconds

❌ 03 Test 4 hours, 2 minutes, 37 seconds

✅ 04 Post-Cleanup 5 seconds

🔄 05 Export-Artifacts

Uh oh!

psap-forge-bot Bot commented Aug 20, 2026

Uh oh!

Harshith-umesh Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

Harshith-umesh commented Aug 20, 2026

Uh oh!

ssaketh-ch commented Aug 20, 2026

Uh oh!

psap-forge-bot Bot commented Aug 20, 2026

Uh oh!

kpouget commented Aug 20, 2026

Uh oh!

Harshith-umesh Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

kpouget Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

ssaketh-ch commented Aug 20, 2026

Uh oh!

ssaketh-ch commented Aug 20, 2026

Uh oh!

psap-forge-bot Bot commented Aug 20, 2026

🟢 Execution of rhaiis nvidia benchmark hera ci-quick 🟢

✅ 00 Pre-Cleanup 2 seconds

✅ 01 Prepare 3 seconds

✅ 02 Preflight 2 seconds

✅ 03 Test 18 minutes, 2 seconds

Test Description

✅ 04 Post-Cleanup 5 seconds

🔄 05 Export-Artifacts

Uh oh!

psap-forge-bot Bot commented Aug 20, 2026

Uh oh!

ssaketh-ch commented Aug 20, 2026

Uh oh!

psap-forge-bot Bot commented Aug 20, 2026

Uh oh!

openshift-ci Bot commented Aug 20, 2026

Uh oh!

psap-forge-bot Bot commented Aug 20, 2026

🟢 Execution of rhaiis nvidia benchmark hera ci-quick 🟢

✅ 00 Pre-Cleanup 2 seconds

✅ 01 Prepare 3 seconds

✅ 02 Preflight 2 seconds

✅ 03 Test 18 minutes, 1 second

Test Description

✅ 04 Post-Cleanup 4 seconds

🔄 05 Export-Artifacts

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

coderabbitai Bot commented Aug 19, 2026 •

edited

Loading

🔴 Execution of `rhaiis` nvidia benchmark hera ci-quick 🔴

✅ 00 Pre-Cleanup `2 seconds`

✅ 01 Prepare `3 seconds`

✅ 02 Preflight `2 seconds`

❌ 03 Test `4 hours, 2 minutes, 37 seconds`

✅ 04 Post-Cleanup `5 seconds`

🟢 Execution of `rhaiis` nvidia benchmark hera ci-quick 🟢

✅ 00 Pre-Cleanup `2 seconds`

✅ 01 Prepare `3 seconds`

✅ 02 Preflight `2 seconds`

✅ 03 Test `18 minutes, 2 seconds`

✅ 04 Post-Cleanup `5 seconds`

🟢 Execution of `rhaiis` nvidia benchmark hera ci-quick 🟢

✅ 00 Pre-Cleanup `2 seconds`

✅ 01 Prepare `3 seconds`

✅ 02 Preflight `2 seconds`

✅ 03 Test `18 minutes, 1 second`

✅ 04 Post-Cleanup `4 seconds`