RFE: add `perf-stat` subtool for hardware performance counter collection

[pradiptapks]

Summary

Add a new perf-stat subtool to tool-kernel that collects hardware performance counter data using perf stat -I <interval> --json. This enables crucible users to capture CPU microarchitecture efficiency metrics (IPC, cache miss rates, branch mispredictions, TLB misses) alongside benchmark execution , data that is critical for understanding why performance changed, not just that it changed.

Motivation

Crucible's existing tool ecosystem captures what the system is doing , CPU utilization (sysstat), interrupt rates (procstat), process activity (forkstat), power consumption (power). But there is no tool that captures how efficiently the CPU is executing work.

Consider a scenario where a uperf run shows 10% lower throughput than a baseline. sysstat shows CPU utilization is the same. Without hardware counter data, the engineer cannot determine whether the regression is caused by:

• Higher cache miss rates (memory subsystem issue)
• More branch mispredictions (code path change)
• TLB pressure (memory mapping issue)
• Lower IPC from pipeline stalls (microarchitecture bottleneck)

perf stat provides exactly this data with minimal overhead (~1-2% CPU).

Why extend tool-kernel rather than create a new tool

1. perf is already built from kernel source in workshop.json , the same binary supports both perf record (existing subtool) and perf stat (proposed). No additional build dependencies.
2. The subtool architecture was designed for this , tool-kernel is explicitly a collection of kernel profiling utilities. perf stat counting is a natural sibling to perf record sampling.
3. Shared parameters , perf stat -I $interval reuses the existing --interval parameter that turbostat already uses.
4. Single deployment , one tool instance handles turbostat,perf-stat on a profiler node instead of deploying a separate tool container.
5. Same privilege requirements , perf stat needs the same CAP_SYS_ADMIN / perf_event_paranoid access as the existing perf record subtool, and tool-kernel already runs on profiler nodes with these privileges.

Proposed metrics

The perf-stat subtool would collect these hardware events and produce CDM metrics:

Metric Name	CDM Class	Derivation
ipc	throughput	instructions / cycles
l1d-cache-miss-rate	throughput	L1-dcache-load-misses / cache-references
l1i-cache-miss-rate	throughput	L1-icache-load-misses / total-instructions (normalized)
llc-miss-rate	throughput	LLC-load-misses / LLC-loads
branch-miss-rate	throughput	branch-misses / branches
dtlb-miss-rate	throughput	dTLB-load-misses / dTLB-loads
itlb-miss-rate	throughput	iTLB-load-misses / iTLB-loads
context-switches-sec	count	context-switches / interval
cpu-migrations-sec	count	cpu-migrations / interval
instructions-sec	throughput	instructions / interval (MIPS)
cycles-sec	throughput	cycles / interval (GHz effective)

Raw counter values would also be available as CDM metrics for custom analysis.

Proposed changes

1. kerneltools-start , new case branch

Add a perf-stat) case in the subtool dispatch loop:

perf-stat)
    echo "Starting perf stat"
    perf_stat_events="${perf_stat_events:-instructions,cycles,cache-references,cache-misses,\
L1-dcache-load-misses,L1-icache-load-misses,\
LLC-load-misses,LLC-loads,\
branch-misses,branches,\
dTLB-load-misses,dTLB-loads,\
iTLB-load-misses,iTLB-loads,\
context-switches,cpu-migrations}"
    cmd="/usr/bin/perf stat -a -I ${interval} --json -e ${perf_stat_events}"
    echo "Going to run: $cmd"
    $cmd > perf-stat-output.json 2>&1 &
    perf_stat_pid=$!
    echo "perf stat pid is $perf_stat_pid"
    echo "$perf_stat_pid" >>kerneltools-pids.txt
    ;;

2. kerneltools-start , new parameter

Add --perf-stat-events to longopts and the parsing loop to allow users to override the default event list:

--perf-stat-events)
    perf_stat_events=$val
    ;;

3. kerneltools-stop , compression case

perf-stat)
    if [ -e perf-stat-output.json ]; then
        echo "Compressing perf stat output"
        ${taskset_cmd} xz --threads=0 perf-stat-output.json
    else
        echo "Warning: perf-stat-output.json was not found"
    fi
    ;;

4. kerneltools-post-process , new file (Python)

Create the post-process script that rickshaw.json already references but does not exist. This script would:

• Read perf-stat-output.json.xz (JSON records from perf stat --json -I)
• Parse each interval record extracting timestamp, event name, and counter value
• Compute derived rates (IPC, miss rates) from raw counter pairs
• Call log_sample() / finish_samples() from toolbox metrics API
• Write postprocess/post-process-data.json

This post-processor would be designed to support future subtools (e.g., turbostat CDM output) via a dispatcher pattern.

5. rickshaw.json , add post-process script to file deployment

{
    "src": "%tool-dir%/kerneltools-post-process",
    "dest": "/usr/bin/"
}

6. Documentation updates

• Update README.md with perf-stat subtool documentation
• Update the subtools comment in kerneltools-start line 21
• Add run-file example

Usage example

"tool-params": [
    {
        "tool": "kernel",
        "params": [
            { "arg": "subtools", "val": "turbostat,perf-stat" },
            { "arg": "interval", "val": "3" }
        ]
    }
]

With custom events (e.g., memory bandwidth on supported platforms):

"tool-params": [
    {
        "tool": "kernel",
        "params": [
            { "arg": "subtools", "val": "perf-stat" },
            { "arg": "interval", "val": "1" },
            { "arg": "perf-stat-events", "val": "instructions,cycles,cache-misses,LLC-load-misses,context-switches,offcore_response.demand_data_rd.any_response" }
        ]
    }
]

Querying results

Once indexed, metrics would be queryable via the standard CDM interface:

crucible get metric --run <id> --source kernel --type ipc
crucible get metric --run <id> --source kernel --type llc-miss-rate

And visible in the CDM web dashboard for overlay with benchmark throughput/latency , enabling direct correlation between microarchitecture efficiency and workload performance.

Platform considerations

• perf stat requires hardware PMU access. On VMs without PMU passthrough, hardware events report <not supported> , perf stat handles this gracefully by omitting unsupported events from output.
• Software events (context-switches, cpu-migrations) work universally, including in VMs and containers.
• The default event list should be validated at start time and reduced to supported events if needed.
• x86_64 and aarch64 share the same generic perf stat interface; architecture-specific events (e.g., Intel uncore, ARM CMN) would be specified via --perf-stat-events.

Acceptance criteria

• perf-stat subtool collects hardware counter data at configurable intervals
• Default event list covers IPC, cache misses (L1d, L1i, LLC), branch mispredictions, TLB misses, context switches, and CPU migrations
• Custom event list supported via --perf-stat-events parameter
• kerneltools-post-process script produces CDM metrics from perf-stat JSON output
• Derived metrics (IPC, miss rates) computed correctly from raw counter pairs
• Metrics queryable via crucible get metric --source kernel --type <metric>
• Metrics visible in CDM web dashboard (compare and deep-dive views)
• Graceful degradation when hardware events are unavailable (VMs, unsupported PMUs)
• Documentation updated (README, start script comment, run-file example)
• Unit tests for post-process script JSON parsing and rate computation

[k-rister]

LGTM. I wonder if as part of this we should rename the existing perf subtool to perf-record so that it is clear what it does. Right now it's slightly ambiguous but it will get worse as we start adding in support for other perf commands like this RFE suggests.

Thoughts @atheurer?