feat: bridge tool multiplex.json into rickshaw-run so it's functionally live - #865
Merged
Conversation
…ly live
Tools have no analog to benchmarks' mv-params/multiplex pipeline today: a
tool's params in a run file are flat and single-valued (schema/tool-
params.json: {"arg", "val"}, no "vals" array, no presets, no regex
validation). A tool authoring a multiplex.json (perftool-incubator/
crucible#653) needs this wiring for the file to do anything at all --
without it, multiplex.json would just be an inert, unread file.
Adds RunState.apply_tool_multiplex(), wired into load_tool_params(): a
tool's flat, single-valued tool-params.json params are wrapped into
multiplex's "sets" input shape (each value becomes a one-element vals
array), run through multiplex.py completely unmodified, and unwrapped
back. Wrapping every value as a singleton makes multiplex's actual
multiplication step (itertools.product over each param's vals) structurally
incapable of producing more than one combination, so tools get real
validation/preset/transform support identical to benchmarks with no risk
of the per-iteration multiplication semantics that don't apply to a tool
(which starts once with a static param set for the run's duration, unlike
a benchmark's per-iteration parameter sweep). No changes needed to
multiplex.py or its own schemas.
A tool opting into multiplex.json without a "defaults" preset that
reproduces its *-start script's own bash defaults will break existing run
files that pass zero/partial params (schema/tool-params.json's "params" key
isn't required) -- covered by a dedicated regression test rather than left
as a docs-only warning.
Verified end-to-end against the real multiplex.py (not just mocked unit
tests) using tool-kernel/tool-sysstat/tool-procstat multiplex.json files
built alongside this change (to be opened as their own PRs, once this
merges, alongside the new tool-metadata.json/benchmark-metadata.json
schemas landing separately in crucible): empty-params defaults path,
explicit valid params, and invalid-param rejection all behave correctly.
This live testing caught a real bug during development --
toolbox.json.save_json_file() unconditionally xz-compresses its output
with no opt-out, which multiplex.py can't read back, so the wrapped-
params file is now written with a plain open()/json.dump() instead,
matching how the existing benchmark mv-params code path already does it.
Refs: perftool-incubator/crucible#653
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
k-rister
force-pushed
the
feat-tool-benchmark-metadata
branch
from
August 12, 2026 13:23
e065ef6 to
b4996d6
Compare
atheurer
approved these changes
Aug 12, 2026
atheurer
left a comment
Contributor
There was a problem hiding this comment.
Approved! Excellent changes with very clear comments and outstanding unit tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of the implementation for perftool-incubator/crucible#653. This PR covers only the rickshaw-side piece: making a tool's
multiplex.jsonfunctionally live. The newtool-metadata.json/benchmark-metadata.jsonschemas (also part of #653, purely descriptive, no rickshaw runtime consumer) are landing separately in the crucible repo instead — see note below.Tools have no analog to benchmarks'
mv-params/multiplex pipeline today: a tool's params in a run file are flat and single-valued (schema/tool-params.json:{"arg", "val"}, novalsarray, no presets, no regex validation). A tool authoring amultiplex.jsonneeds this wiring for the file to do anything at all — without it,multiplex.jsonwould just be an inert, unread file.RunState.apply_tool_multiplex()(new, wired intoload_tool_params()): a tool's flat, single-valuedtool-params.jsonparams are wrapped into multiplex'ssetsinput shape (each value becomes a one-elementvalsarray), run throughmultiplex.pycompletely unmodified, and unwrapped back.itertools.productover each param'svals) structurally incapable of producing more than one combination — so tools get real validation/preset/transform support identical to benchmarks, with no risk of the per-iteration multiplication semantics that don't apply to a tool (which starts once with a static param set for the run's duration, unlike a benchmark's per-iteration sweep). No changes needed tomultiplex.pyor its own schemas.Backward-compatibility note
schema/tool-params.jsondoesn't require aparamskey — a tool entry with zero params is legal today and means "use the*-startscript's own bash defaults." A tool opting intomultiplex.jsonmust include adefaultspreset reproducing those bash defaults, or existing run files that pass zero/partial params will start hard-failing (EC_EMPTY_SET_FAIL). Covered by a dedicated regression test, not just a docs note.Verification
pytest -v tests/*.py— 21/21 passing (15 pre-existing + 6 newapply_tool_multiplextests, mockingrun_cmdsince CI doesn't check outmultiplex)multiplex.py(not just mocked unit tests) usingtool-kernel/tool-sysstat/tool-procstatmultiplex.jsonfiles built alongside this work (to be opened as their own PRs once this merges, alongside thetool-metadata.json/benchmark-metadata.jsonschemas landing in crucible): empty-params defaults path, explicit valid params, and invalid-param rejection all verified correcttoolbox.json.save_json_file()unconditionally xz-compresses its output with no opt-out, whichmultiplex.pycan't read back — fixed by writing the wrapped-params file with a plainopen()/json.dump()instead, matching how the existing benchmarkmv-paramscode path already does itSequencing note
Once this merges: the new
tool-metadata.json/benchmark-metadata.jsonschemas land in crucible'sschema/(their only planned consumer is crucible#654's futurecrucible tools listcommand), andtool-kernel/tool-sysstat/tool-procstat/bench-fioreference-implementation PRs open in their own repos — their CI checks out rickshaw's live default branch, not a paired PR branch, so it can't exercise this wiring until it's onmaster.Refs: perftool-incubator/crucible#653
🤖 Generated with Claude Code