Skip to content

feat: bridge tool multiplex.json into rickshaw-run so it's functionally live - #865

Merged
k-rister merged 1 commit into
masterfrom
feat-tool-benchmark-metadata
Aug 12, 2026
Merged

feat: bridge tool multiplex.json into rickshaw-run so it's functionally live#865
k-rister merged 1 commit into
masterfrom
feat-tool-benchmark-metadata

Conversation

@k-rister

@k-rister k-rister commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Part of the implementation for perftool-incubator/crucible#653. This PR covers only the rickshaw-side piece: making a tool's multiplex.json functionally live. The new tool-metadata.json/benchmark-metadata.json schemas (also part of #653, purely descriptive, no rickshaw runtime consumer) are landing separately in the crucible repo instead — see note below.

Tools have no analog to benchmarks' mv-params/multiplex pipeline today: a tool's params in a run file are flat and single-valued (schema/tool-params.json: {"arg", "val"}, no vals array, no presets, no regex validation). A tool authoring a multiplex.json needs this wiring for the file to do anything at all — without it, multiplex.json would just be an inert, unread file.

  • RunState.apply_tool_multiplex() (new, wired into load_tool_params()): a tool's flat, single-valued tool-params.json params are wrapped into multiplex's sets input shape (each value becomes a one-element vals array), run through multiplex.py completely unmodified, and unwrapped back.
  • Wrapping every value as a singleton makes multiplex's actual multiplication step (itertools.product over each param's vals) structurally incapable of producing more than one combination — so tools get real validation/preset/transform support identical to benchmarks, with no risk of the per-iteration multiplication semantics that don't apply to a tool (which starts once with a static param set for the run's duration, unlike a benchmark's per-iteration sweep). No changes needed to multiplex.py or its own schemas.

Backward-compatibility note

schema/tool-params.json doesn't require a params key — a tool entry with zero params is legal today and means "use the *-start script's own bash defaults." A tool opting into multiplex.json must include a defaults preset reproducing those bash defaults, or existing run files that pass zero/partial params will start hard-failing (EC_EMPTY_SET_FAIL). Covered by a dedicated regression test, not just a docs note.

Verification

  • pytest -v tests/*.py — 21/21 passing (15 pre-existing + 6 new apply_tool_multiplex tests, mocking run_cmd since CI doesn't check out multiplex)
  • End-to-end against the real multiplex.py (not just mocked unit tests) using tool-kernel/tool-sysstat/tool-procstat multiplex.json files built alongside this work (to be opened as their own PRs once this merges, alongside the tool-metadata.json/benchmark-metadata.json schemas landing in crucible): empty-params defaults path, explicit valid params, and invalid-param rejection all verified correct
  • This live testing caught a real bug during development: toolbox.json.save_json_file() unconditionally xz-compresses its output with no opt-out, which multiplex.py can't read back — fixed by writing the wrapped-params file with a plain open()/json.dump() instead, matching how the existing benchmark mv-params code path already does it

Sequencing note

Once this merges: the new tool-metadata.json/benchmark-metadata.json schemas land in crucible's schema/ (their only planned consumer is crucible#654's future crucible tools list command), and tool-kernel/tool-sysstat/tool-procstat/bench-fio reference-implementation PRs open in their own repos — their CI checks out rickshaw's live default branch, not a paired PR branch, so it can't exercise this wiring until it's on master.

Refs: perftool-incubator/crucible#653

🤖 Generated with Claude Code

…ly live

Tools have no analog to benchmarks' mv-params/multiplex pipeline today: a
tool's params in a run file are flat and single-valued (schema/tool-
params.json: {"arg", "val"}, no "vals" array, no presets, no regex
validation). A tool authoring a multiplex.json (perftool-incubator/
crucible#653) needs this wiring for the file to do anything at all --
without it, multiplex.json would just be an inert, unread file.

Adds RunState.apply_tool_multiplex(), wired into load_tool_params(): a
tool's flat, single-valued tool-params.json params are wrapped into
multiplex's "sets" input shape (each value becomes a one-element vals
array), run through multiplex.py completely unmodified, and unwrapped
back. Wrapping every value as a singleton makes multiplex's actual
multiplication step (itertools.product over each param's vals) structurally
incapable of producing more than one combination, so tools get real
validation/preset/transform support identical to benchmarks with no risk
of the per-iteration multiplication semantics that don't apply to a tool
(which starts once with a static param set for the run's duration, unlike
a benchmark's per-iteration parameter sweep). No changes needed to
multiplex.py or its own schemas.

A tool opting into multiplex.json without a "defaults" preset that
reproduces its *-start script's own bash defaults will break existing run
files that pass zero/partial params (schema/tool-params.json's "params" key
isn't required) -- covered by a dedicated regression test rather than left
as a docs-only warning.

Verified end-to-end against the real multiplex.py (not just mocked unit
tests) using tool-kernel/tool-sysstat/tool-procstat multiplex.json files
built alongside this change (to be opened as their own PRs, once this
merges, alongside the new tool-metadata.json/benchmark-metadata.json
schemas landing separately in crucible): empty-params defaults path,
explicit valid params, and invalid-param rejection all behave correctly.
This live testing caught a real bug during development --
toolbox.json.save_json_file() unconditionally xz-compresses its output
with no opt-out, which multiplex.py can't read back, so the wrapped-
params file is now written with a plain open()/json.dump() instead,
matching how the existing benchmark mv-params code path already does it.

Refs: perftool-incubator/crucible#653

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@k-rister
k-rister force-pushed the feat-tool-benchmark-metadata branch from e065ef6 to b4996d6 Compare August 12, 2026 13:23
@k-rister k-rister changed the title feat: add tool-metadata.json/benchmark-metadata.json schemas and live tool multiplex support feat: bridge tool multiplex.json into rickshaw-run so it's functionally live Aug 12, 2026

@atheurer atheurer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved! Excellent changes with very clear comments and outstanding unit tests.

@k-rister
k-rister merged commit 918eb2e into master Aug 12, 2026
402 of 417 checks passed
@k-rister
k-rister deleted the feat-tool-benchmark-metadata branch August 12, 2026 15:25
@k-rister
k-rister restored the feat-tool-benchmark-metadata branch August 12, 2026 15:25
@k-rister
k-rister deleted the feat-tool-benchmark-metadata branch August 12, 2026 15:25
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Crucible Tracking Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants