Skip to content

mpstat: add 'node' (NUMA node) and 'cpu' (alias for num) as CDM breakout dimensions #68

Description

@atheurer

Background

When querying mpstat data via the CDM API, the Linux CPU number is exposed as the breakout dimension num. This works, but creates two usability gaps:

  1. No node breakout — There is no way to directly query per-NUMA-node CPU utilization. Analysts must manually cross-reference num values against the host's CPU topology to determine which NUMA node a busy CPU belongs to. On large systems (e.g. 768-CPU AMD EPYC with 2 NUMA nodes), this is cumbersome.

  2. Non-obvious dimension name — The dimension is called num rather than cpu. Users and agents familiar with the concept of "CPU number" naturally try cpu as a breakout name and get no results, then incorrectly conclude that per-CPU data is not available.

Requested Changes

1. Add cpu as an alias for num

Allow --breakout cpu to work identically to --breakout num — same data, just a more intuitive name. Keeping num for backwards compatibility.

2. Add node as a derived breakout

Derive the NUMA node for each CPU using sysfs topology (/sys/devices/system/node/nodeN/cpulist or /sys/devices/system/cpu/cpuN/node*) at collection time and store it as a breakout dimension. This allows queries like:

crucible get metric --period <ID> --source mpstat --type Busy-CPU --breakout hostname,node

to immediately show per-NUMA-node aggregate CPU utilization without manual CPU-to-node mapping.

Other potentially useful topology breakouts to consider

While addressing the above, the following sysfs-derivable dimensions would also be valuable for network performance analysis:

  • shared-l3-domain — grouping CPUs by last-level cache domain (useful for identifying cache-sharing patterns between application and interrupt-handling threads)
  • die — already available as a CDM breakout but worth confirming correct derivation on multi-die platforms

Context

This came up during a 400G TCP stream benchmark investigation where:

  • The NIC's NUMA node was node 1 (CPUs 192-383, 576-767)
  • IRQs were correctly pinned to CPUs 192-223
  • We needed to verify NUMA locality of application threads vs. IRQ-handling CPUs
  • Querying by num and then manually checking which CPUs are in node 1 vs node 0 added friction

A node breakout would allow a one-line query to confirm NUMA locality rather than a multi-step manual cross-reference.

AI-assisted-by: Claude Sonnet 4.6

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions