** [BUG] perf-stat::ipc ratio aggregation returns corrupted negative values in CDM queries

[atheurer]

🐛 Issue 1 (tool-kernel): perf-stat::ipc Ratio Aggregation Bug

• Repository: perftool-incubator/tool-kernel
• Title: [BUG] perf-stat::ipc ratio aggregation returns corrupted negative values in CDM queries
• Labels: bug, CDM-integration

📝 Problem Description

When querying the perf-stat::ipc (Instructions Per Cycle) metric through the Common Data Model (CDM) REST API endpoint /api/v1/metric-data over steady-state periods, the database returns corrupted, physically impossible negative float values:

// Example corrupted response for CPU 43 on host f11-h18
"<f11-h18-000-r6625.rdu2.scalelab.redhat.com>-<43>": [
  {"begin": 1785440466944, "end": 1785440646322, "value": -2886859.536809956}
]

These corrupted ratios break analytical agents, forcing them to execute expensive manual shell commands and custom Python parser fallbacks to scan raw XZ logs.

🔍 Technical Root Cause

The bug resides in how ipc is pre-calculated per interval and exported to CDM inside kerneltools-post-process.py:

if cycles > 0 and instructions > 0:
    ipc = instructions / cycles
    metrics.log_sample(
        SOURCE_PERF_STAT,
        {"source": SOURCE_PERF_STAT, "class": "throughput", "type": "ipc"},
        names,
        {**sample_base, "value": ipc},
    )

1. Ratio Summation/Mean Error: ipc is computed and stored as a pre-divided ratio per interval (instructions / cycles).
2. Database Aggregation Failure: When the CDM database aggregates these metrics over a multi-minute period, it applies standard SUM or AVG arithmetic aggregation on these pre-divided ratios. This is mathematically invalid and triggers severe overflow/division anomalies in SQL.

💡 Suggested Solution (Coordinated with CDM PR #202)

To correctly aggregate ratio metrics like IPC, CDM needs to track numerator (instructions) and denominator (cycles) variables independently, and divide them at query-time rather than pre-calculating them at post-process time.

We must leverage CDM PR #202 (CommonDataModel/pull/202), which introduces new aggregation options for CDM:

1. Update kerneltools-post-process.py to utilize these new aggregation options for perf-stat metrics.
2. Define a custom ratio aggregation schema for ipc where the database sums instructions and cycles independently over the period, and computes sum(instructions) / sum(cycles) dynamically at query time.