Skip to content

Add golden-file regression test for sysstat-post-process #62

Description

@atheurer

Background

Post-processor scripts convert collected system data to CDM metrics via
`CDMMetrics.log_sample()`, writing `metric-data-*.{csv,json}.xz` output.
Without a regression test, changes to the post-processor (optimizations,
new metrics, refactors) can silently produce wrong output.

This issue tracks adding a golden-file regression test for sysstat-post-process.

What to build

```
test/
data/ # committed sample collector output (real, anonymized)
sar/
sar-.xz # sar output files # tool-specific input files
golden/ # committed known-good post-processor output
metric-data-
.{csv,json}.xz # generated once from trusted code, committed
run-test.sh # run post-processor on data/, compare to golden/
make-golden.sh # (re)generate golden/ from data/ — run manually, never by CI
```

Test data

Collect a small but realistic sample by running the tool against a real system
for a short interval (30–60 seconds is sufficient). Truncate to a small time
range if needed. The data should:

  • Exercise the normal code paths (non-trivial, non-degenerate input)
  • Be small enough to commit to git (target < 2 MB uncompressed per input file)
  • Not contain sensitive hostnames or credentials

run-test.sh

```bash
#!/bin/bash
set -e
SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
TMPDIR=$(mktemp -d)
trap "rm -rf $TMPDIR" EXIT

cp -r "$SCRIPT_DIR/data/." "$TMPDIR/"
(cd "$TMPDIR" && TOOLBOX_HOME=... python3 "$SCRIPT_DIR/../sysstat-post-process.py")

python3 -m toolbox.test_cdm_output "$TMPDIR" "$SCRIPT_DIR/golden/"
```

Comparison semantics (handled by toolbox.test_cdm_output)

  • Same metric type sets: every `(desc.type, names)` tuple matches, compared
    by name not by idx (idx is an internal implementation detail).
  • Same non-zero values: every non-zero CSV row in golden appears in the
    candidate at the same `(metric_name, end_timestamp, value)`.
  • Tolerate leading zeros: zero-value rows before a metric's first active
    period may be absent from the candidate; this is acceptable.

make-golden.sh

Run manually when output intentionally changes (new metric, renamed field).
Commits the updated golden. CI never regenerates golden — only compares.

Acceptance criteria

  • `test/data/` committed with realistic sample input
  • `test/golden/` committed with output from current (trusted) post-processor
  • `test/run-test.sh` exits 0 when output matches golden, non-zero on mismatch
  • `test/make-golden.sh` regenerates golden from data
  • CI runs `test/run-test.sh` on every PR (add to existing workflow or new one)

Dependencies

Requires toolbox#127 (shared `test_cdm_output.py` comparison utility).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Queued

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions