Skip to content

[rhaiis] Add missing workload profiles 5-7 and composite benchmark presets #181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hp2419 wants to merge 1 commit into from
Closed

[rhaiis] Add missing workload profiles 5-7 and composite benchmark presets #181

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions projects/rhaiis/orchestration/presets.d/benchmarks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,39 @@ benchmark:
rhaiis.profiler.enabled: true
caliper.postprocess.csv_dashboard.enabled: true
rhaiis.agent_analysis.enabled: false

benchmark-standard:
extends:
- benchmark
tests.rhaiis.workload_key:
- profile1
- profile2

benchmark-long-context:
extends:
- benchmark
tests.rhaiis.workload_key:
- profile4
- profile5

benchmark-full-sweep:
extends:
- benchmark
tests.rhaiis.workload_key:
- profile1
- profile2
- profile3
- profile4
- profile5
- profile6
- profile7

benchmark-multi-turn:
extends:
- benchmark
tests.rhaiis.workload_key: profile6

benchmark-heterogeneous:
extends:
- benchmark
tests.rhaiis.workload_key: profile7
Comment on lines +36 to +44

@kpouget kpouget Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

random question from a rhaiis-outsider: is there no way to give a meaningful name to the profiles?
profile6 seems to be multi-turn and profile7 is heterogeneous,

but are these number profiles mandatory? 🤔

@hp2419 hp2419 Aug 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great question! The numbered names (profile1–7) are inherited from model-furnace, where they're used in regression comparison (PROFILE_MAP), payload pattern matching
(_PROFILE_PATTERNS), and benchmark launch scripts. Renaming them in forge would break the alignment between the two systems and make cross-system result comparisons harder.

That said, the preset layer is exactly where we add meaningful names. The workload keys stay as profile1–profile7, but users interact through presets like benchmark-multi-turn,
benchmark-heterogeneous, benchmark-long-context, etc.

We could also consider adding aliases in workloads.yaml (e.g. multi-turn pointing to the same config as profile6) if we want meaningful names at the workload level too, without breaking the numbered convention.

Loading