feat: add perf-stat subtool for per-CPU IPC measurement - #59
Open
atheurer wants to merge 3 commits into
Open
Conversation
Add kerneltools-post-process.py registered in rickshaw.json as controller.post-script. It dispatches based on which output files exist in the tool data directory, so it handles gracefully when subtools other than turbostat are used. Turbostat handler parses the TSV output produced by kerneltools-start with -d -d (debug) flags and emits: System aggregate (Package/CPU = "-"): cpu-busy-pct, cpu-freq-avg-mhz, package-power-watt Per-CPU (with cpu breakout): cpu-busy-pct, cpu-busy-freq-mhz, c1-pct, c2-pct, ipc Tested against real PERF-E0306461 run data: 478 samples, 643 metric types, correct timestamps and values verified. AI-assisted-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Adds perf-stat as a new subtool alongside turbostat. Uses `perf stat -a -A -I <interval_ms> -x , -e cycles,instructions,cache-misses,LLC-load-misses` to report hardware performance counters per CPU at configurable intervals. IPC (instructions/cycles) per CPU per interval is the primary metric — it reveals whether throughput variability is caused by more work (same IPC, more cycles) or by execution stalls (lower IPC, same cycles). This distinguishes cache/memory pressure from actual workload variation. Metrics emitted to CDM: perf-stat:ipc per-CPU throughput perf-stat:cache-miss-rate per-CPU utilization (%) perf-stat:llc-load-miss-rate per-CPU utilization (%) Usage: --subtools perf-stat --subtools turbostat,perf-stat --interval 3 (3-second reporting intervals) Requires: bare-metal host with hardware PMU access (not VMs). debugfs is auto-mounted if not already present. AI-assisted-by: Claude Sonnet 4.6
tc-wilson
previously approved these changes
Jul 28, 2026
LLC-load-misses is not supported on AMD EPYC (returns <not supported>).
Replace with stalled-cycles-backend and stalled-cycles-frontend which
are generic hardware aliases that work on both Intel and AMD:
AMD Zen 4: stalled-cycles-backend = event=0x87,umask=0x01
stalled-cycles-frontend = event=0x87,umask=0x02
These metrics directly answer the memory-bound vs core-bound question:
backend-stall-rate = stalled-cycles-backend / cycles * 100
frontend-stall-rate = stalled-cycles-frontend / cycles * 100
A high backend-stall-rate with low cache-miss-rate (as observed on AMD
EPYC Genoa) points to memory ordering stalls, TLB pressure, or store
buffer conflicts — not L3 cache capacity misses.
AI-assisted-by: Claude Sonnet 4.6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
perf-statas a new subtool alongsideturbostatandperf. Usesperf stat -a -A -I <interval_ms> -x ,to report hardware performance counters per CPU at configurable intervals, and post-processes the output into CDM metrics.Motivation
When investigating throughput variability in single-threaded uperf TCP stream tests, we found that CPU utilization on the sender's pinned CPU is sometimes 100% sys — but we can't tell why from mpstat alone. IPC (instructions per cycle) per CPU at 3-second intervals would reveal:
This is the key diagnostic that mpstat and turbostat don't provide.
Metrics emitted to CDM
perf-stat:ipc{cpu: N}perf-stat:cache-miss-rate{cpu: N}perf-stat:llc-load-miss-rate{cpu: N}Usage
Notes
🤖 Generated with Claude Code