Add machine-readable tool metadata: tool-metadata.json and multiplex.json for all tools

[atheurer]

Problem

Crucible benchmarks have multiplex.json for parameter definitions (presets, validations), but tools have no equivalent metadata. Tool parameters, subtool declarations, and descriptions exist only as bash comments and getopt strings inside *-start scripts — not machine-readable.

This makes it impossible for automation (including AI agents) to discover:

• What tools exist and what they do
• Which tools have subtools (and what those subtools are)
• What parameters each tool accepts and their valid values
• Which tools/subtools produce CDM-indexed metrics vs raw file output only

Current state

File	What it has	What it's missing
rickshaw.json	Tool name, start/stop scripts, endpoint whitelist/blacklist	Description, params, subtools
workshop.json	Build dependencies (packages, source tarballs)	Everything semantic
*-start scripts	Params as bash getopt longopts; subtool lists as bash comments	Not machine-readable
multiplex.json	Does not exist for tools (only benchmarks)	Everything

Tools with subtools today

Discovered by reading bash scripts — this should be in metadata:

• kernel → turbostat, perf, speed-select-util, trace-cmd, sysfs-trace (params: --subtools, --interval, --record-opts, --base-freq, --turbo-freq, --core-power, --trace-cmd-record-opts, --sysfs-trace-setup, --sysfs-trace-cleanup)
• sysstat → mpstat, sar, iostat, pidstat (params: --subtools, --interval)
• ethtool — no subtools (params: --interval, --interfaces)
• forkstat — no subtools (params: --events)
• ftrace — no subtools (params: --record-opts)
• mlxreg — no subtools (params: --devices, --sensors, --interval)
• nvidia — no subtools (params: --interval)
• ovs — no subtools (params: --interval)
• power — no subtools (params: --interval, --username, --password, --endpoints, --plugin)
• procstat — no subtools (params: --files, --interval)
• rt-trace-bpf — no subtools (params: --cpu-list)

Proposed changes

1. Add tool-metadata.json to each tool subproject

New file alongside rickshaw.json and workshop.json:

{
  "tool": "kernel",
  "description": "Linux kernel performance tools — CPU frequency, power, hardware counters, and tracing",
  "subtools": [
    {
      "name": "turbostat",
      "description": "CPU frequency, power, and thermal monitoring via MSR polling",
      "cdm_indexed": false,
      "output_files": ["turbostat-stdout.txt"]
    },
    {
      "name": "perf",
      "description": "Hardware performance counter sampling (IPC, cache misses, branch stats)",
      "cdm_indexed": false,
      "output_files": ["perf-stat-stdout.txt"]
    },
    {
      "name": "speed-select-util",
      "description": "Intel Speed Select Technology configuration and monitoring",
      "cdm_indexed": false,
      "output_files": ["settings.txt"]
    },
    {
      "name": "trace-cmd",
      "description": "Ftrace-based kernel event tracing",
      "cdm_indexed": false,
      "output_files": ["trace.dat"]
    },
    {
      "name": "sysfs-trace",
      "description": "Sysfs attribute polling for hardware-specific counters",
      "cdm_indexed": false,
      "output_files": []
    }
  ]
}

For tools without subtools:

{
  "tool": "procstat",
  "description": "Per-process and per-CPU interrupt and softirq statistics from /proc",
  "cdm_indexed": true,
  "cdm_sources": [
    {"source": "procstat", "types": ["interrupts-sec", "processed-sec"]}
  ]
}

The cdm_indexed and cdm_sources fields tell consumers which metrics will appear in CDM after post-processing, and which will only be available as raw files.

2. Add multiplex.json to each tool subproject

Same format as benchmark multiplex.json:

{
  "presets": {
    "basic": [
      {"arg": "subtools", "vals": ["turbostat"]},
      {"arg": "interval", "vals": ["10"]}
    ]
  },
  "validations": {
    "subtools": {
      "description": "comma-separated list of kernel subtools to run",
      "args": ["subtools"],
      "vals": "^(turbostat|perf|speed-select-util|trace-cmd|sysfs-trace)(,(turbostat|perf|speed-select-util|trace-cmd|sysfs-trace))*$"
    },
    "positive_integer": {
      "description": "a whole number greater than 0",
      "args": ["interval"],
      "vals": "[1-9][0-9]*"
    }
  }
}

3. Schema validation

Add JSON schemas for both new files under schema/ so they can be validated alongside existing rickshaw.json and workshop.json schemas.

Files to add/modify

Per tool subproject (11 tools):

• Add tool-metadata.json
• Add multiplex.json

Crucible core:

• schema/tool-metadata.json — JSON schema for validation
• schema/multiplex.json — JSON schema (may already exist for benchmarks)

[k-rister]

Started implementation, opened as rickshaw PR #865: perftool-incubator/rickshaw#865

Scope was expanded during design, worth noting here for anyone following this issue:

1. Benchmarks get the same treatment. They have the identical gap (no description/CDM-manifest metadata today), so a parallel benchmark-metadata.json schema was added alongside tool-metadata.json, using sub-benchmarks instead of subtools (no current benchmark is a suite of sub-benchmarks, but the schema doesn't foreclose one being implemented that way someday).
2. Corrected schema location. The new schemas live in rickshaw/schema/ (next to the existing tool.json/benchmark.json), not crucible's own schema/ — crucible's schema/ only validates crucible's own config files (registries.json/repos.json/services.json).
3. multiplex.json for tools is fully live, not just descriptive. Tools previously had no analog at all to benchmarks' multiplex pipeline. rickshaw PR #865 adds RunState.apply_tool_multiplex(), which runs a tool's params through the real, unmodified multiplex.py for validation/presets/transforms — with a safety property that makes it impossible for a tool's params to ever get multiplied into more than one iteration (unlike benchmarks, a tool starts once with a static param set for the whole run).
4. Corrected kernel's subtool name. The survey in this issue's description listed speed-select-util; the actual bash case statement in kerneltools-start uses intel-speed-select.

Sequencing: reference implementations for tool-kernel, tool-sysstat, tool-procstat, and bench-fio were built and verified locally against this PR's branch, and will be opened as their own PRs once rickshaw #865 merges (their CI checks out rickshaw's live default branch, not a paired PR branch, so it can't exercise the new wiring until then).

[k-rister]

Correction to my previous comment: the `tool-metadata.json`/`benchmark-metadata.json` schemas are moving out of rickshaw PR #865, not staying in it.

Reasoning: nothing in rickshaw actually loads or validates against these two schemas (confirmed by grep — zero references in `rickshaw-run.py`). They were initially placed in `rickshaw/schema/` by analogy to `tool.json`/`benchmark.json` (which genuinely are validated by `rickshaw-run.py` at runtime), but that's the wrong criterion. There's already a precedent in this exact area for the right one: `multiplex.json` instance files also live in tool/benchmark subprojects, and rickshaw also just orchestrates calling `multiplex.py` — but the schema that validates `multiplex.json` (`req-schema.json`) lives in the `multiplex` repo itself, because `multiplex.py` is what actually runs `jsonschema.validate()` against it.

The only planned consumer of `tool-metadata.json`/`benchmark-metadata.json` is this issue's own #654 (`crucible tools list`), which lives in the crucible repo — so the schemas belong in crucible's `schema/`, not rickshaw's. They'll land there as a separate crucible-side PR once rickshaw PR #865 (now scoped to just the tool-multiplex wiring) merges.