diff --git a/CHANGELOG.md b/CHANGELOG.md index ccf71f1..a4747a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,13 @@ All notable changes to this project are documented here. Format follows [Keep a ## [Unreleased] +### Added: one-command verdict and calibrated baselines (production-QA direction) + +- **Verdict banner.** `./run` now ends with a clear `SHAKEDOWN VERDICT: PASS/FAIL` banner (green/red on a TTY) summarizing the performance result, so a run finishes with an unambiguous answer instead of a JSON dump. Performance-focused (the batch-variance / thermal / memory / GPU defect class), not cosmetics. +- **Calibrated baselines.** `make-baseline.sh` turns N known-good reports into `baselines/.json` (per-metric golden value plus a 3-sigma control limit, or a tolerance band below 8 samples). When `baselines/.json` exists for the `--target`, `./run` bins each metric against its golden limit and a metric outside its limit escalates the result to FAIL: the verdict becomes a calibrated pass/fail instead of within-unit advisory, recorded in the new optional `baseline_check` field. Absent a baseline, behavior is unchanged. Schema 1.4. +- **`Verification/Production QA.md`.** An honest engineering gap analysis between Shakedown today and a factory production-QA station (calibrated limits, golden binning, Measurement System Analysis, gated preconditions, defect coverage / escape rate, SPC), and the architecture to close it. The calibrated-baseline mechanism is the first leg. +- **Store Day Checklist** reframed to a single in-store flow centered on the one command (`./run --store`) and the verdict banner. + ### Added: in-store verification toolkit (wave 8) - **`--store` thorough profile.** One flag for verifying a new unit (M5 especially): turns on `--noaccel` and `--gpu` and runs a longer warmup with more iterations, so an intermittent batch defect has more chances to surface. Honors any timing env vars the user set. @@ -12,6 +19,7 @@ All notable changes to this project are documented here. Format follows [Keep a - **Phase 12 STREAM triad.** Memory bandwidth now prefers a vendored single-file STREAM triad (`stream-triad.c`: real copy / scale / add / triad), compiled at runtime with clang, no network. Falls back to the pure-Python `memmove` proxy when clang is unavailable. Closes the copy-only-proxy gap; `details.method` records which ran. - **Phase 14 `--llama` (opt-in).** Clones and builds llama.cpp at a pinned ref and runs `llama-bench`, a combined CPU+GPU+memory AI load (the workload class the M5 defect was reported on). Off by default; it reaches the network and runs third-party code (disclosed in SECURITY.md) and skips cleanly without git / cmake / network / model. - **Schema 1.3.** Phase 12 gains the triad method and `*_triad_gb_per_s` fields; new phase key `14_llama_bench` (a `skipped` placeholder by default), added to the CI submission-audit required-phase list. Backward compatible: both phase-12 methods keep `mean_copy_gb_per_s`, and 14 is skipped unless opted in. +- **Store Day Checklist** (`Verification/Store Day Checklist.md`). A short, top-to-bottom run-this-then-that guide for verifying a unit at the store and then at the hotel, sitting on top of the Benchmark Reference. Linked first from the README and the Benchmark Reference. ### Added: roadmap completion (wave 7) diff --git a/README.md b/README.md index 28a3619..0279209 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ Or without a preset, which auto-detects chassis class from `system_profiler` (fi The orchestrator runs the automated phases (preflight → inventory → battery → race benchmark → SSD test → memory bandwidth → CPU variance → thermal load) end-to-end, asks for sudo once upfront (Phase 5 and the SSD page-cache drop need it), and writes a SCHEMA-compliant report to `Reports/local/` plus a sanitized PR-able copy to `Reports/submissions/`. Opt-in flags add heavier passes: `--noaccel` (a non-accelerated BLAKE2b variance pass), `--gpu` (a Metal GPU compute pass), and `--llama` (clones and builds llama.cpp for a combined CPU+GPU+memory AI load). `--store` bundles the thorough profile for verifying a new unit. Runtime ~20 min on Intel, ~27 min on Air, ~47 min on MacBook Pro. -> Verifying a new purchase, especially in a store you can't easily return to? The [Benchmark Reference](Verification/Benchmark%20Reference.md) is the install-this / run-this / expected-score guide (sourced per-generation Cinebench + Geekbench baselines, the in-store and hotel protocols, and crowd-sourced live-lookup links). To diff your unit against a known-good sibling, run `Verification/scripts/compare-reports.sh reference.json yours.json`. +> Verifying a new purchase, especially in a store you can't easily return to? Follow the [Store Day Checklist](Verification/Store%20Day%20Checklist.md), a top-to-bottom run-this-then-that guide for verifying your machine in one sitting at the store. The [Benchmark Reference](Verification/Benchmark%20Reference.md) behind it has the sourced per-generation expected scores and live-lookup links, and `Verification/scripts/compare-reports.sh reference.json yours.json` diffs your unit against a known-good sibling. `./run --no-sudo` skips the 10-min thermal phase (the only phase that needs sudo) for a half-runtime no-password variance-only pass. @@ -118,6 +118,8 @@ mac-shakedown/ │ ├── Runbook.md # the procedure │ ├── Pass-Fail Criteria.md # thresholds (parameterized by chassis class) │ ├── Benchmark Reference.md # install/run/expected-score + in-store protocol +│ ├── Store Day Checklist.md # the one-command in-store flow +│ ├── Production QA.md # gap analysis: toward factory-grade QA │ ├── per-core-pinning.md # why macOS has no per-core affinity (methodology note) │ └── scripts/ │ ├── run-shakedown.sh # the orchestrator (`./run` execs this) @@ -132,7 +134,9 @@ mac-shakedown/ │ ├── gpu-variance.sh # opt-in Metal compute variance, swiftc at runtime → JSON │ ├── llama-bench.sh # opt-in llama.cpp combined load (clone+build) → JSON │ ├── compare-reports.sh # diff two reports (unit vs known-good sibling) +│ ├── make-baseline.sh # build baselines/.json from known-good reports │ └── display-test.sh # fullscreen color cycle (HTML) +├── baselines/ # calibrated golden limits per SKU (mostly empty; built from known-good runs) ├── targets/ # preset SKU configs │ ├── README.md │ ├── mbp-16-m5-max-64.json @@ -190,10 +194,12 @@ Wave 7 shipped most of the original roadmap: the non-accelerated variance pass ( Wave 8 added the in-store toolkit: the `--store` thorough profile, `compare-reports.sh` (diff your unit against a known-good sibling), a sourced [Benchmark Reference](Verification/Benchmark%20Reference.md), a vendored STREAM triad in Phase 12 (real copy / scale / add / triad), and an opt-in `--llama` phase that builds llama.cpp for a combined CPU+GPU+memory load. +Since then: a one-command verdict banner on `./run`, a single-sitting [Store Day Checklist](Verification/Store%20Day%20Checklist.md), and the first leg of production-grade QA, calibrated golden baselines (`make-baseline.sh` builds `baselines/.json` from known-good runs; `./run` then bins against it for a calibrated verdict). [`Verification/Production QA.md`](Verification/Production%20QA.md) is the honest gap analysis to a factory-grade screen. + Still open: - **Hosted aggregator.** Eventually, submission via API to a public site so reports aren't reviewed by hand. Until then, the PR-submission flow above *is* the aggregator. Slower, but no infra, and PR review catches PII before merge. -- **Calibrated v0.3 thresholds.** The race / SSD / memory-bandwidth / GPU phases are informational in v0.2. Once the submission corpus has per-SKU baselines, they get chassis-family pass/fail bands. The SSD floor is the only band today, and it's a conservative sanity floor (flags below 500 MB/s with the cache dropped), not a calibrated range. +- **A corpus to calibrate against.** The mechanism now exists: `./run` bins against a golden baseline when `baselines/.json` is present (`make-baseline.sh` builds one from known-good reports), turning the verdict calibrated. What is still missing is the population of known-good units to characterize, plus a Measurement System Analysis (is our gage error small vs the tolerance?) and gated preconditions. [`Verification/Production QA.md`](Verification/Production%20QA.md) lays out the full path to a production-grade screen. - **Per-core pinning.** macOS lacks public CPU affinity APIs, so we can't pin workers to specific cores; a defective single core gets averaged across N P-cores. Reporting `worker_imbalance_pct_per_iter` is the partial mitigation. The investigation is written up in [`Verification/per-core-pinning.md`](Verification/per-core-pinning.md): the short version is that `THREAD_AFFINITY_POLICY` is ignored on Apple Silicon and QoS hints only steer between the P and E clusters, so there's no per-core pinning to be had today. ## Origin diff --git a/Reports/SCHEMA.md b/Reports/SCHEMA.md index 405e0d4..639367d 100644 --- a/Reports/SCHEMA.md +++ b/Reports/SCHEMA.md @@ -9,7 +9,7 @@ Every QA run produces two artifacts in `Reports/`: | Field | Type | Required | Notes | |---|---|---|---| -| `schema_version` | string | yes | Semver of the schema. Currently `"1.3"`. | +| `schema_version` | string | yes | Semver of the schema. Currently `"1.4"`. | | `shakedown_version` | string | yes | Tool version that produced the report (e.g. `"0.1.0"`). | | `timestamp` | string | yes | ISO 8601 UTC. | | `illustrative` | bool | no | `true` for hand-crafted samples (under `examples/sample-report-*`). Aggregator must filter these out. | @@ -19,6 +19,7 @@ Every QA run produces two artifacts in `Reports/`: | `phases` | object | yes | One key per phase (see Phase shape). | | `result` | string | yes | `"PASS"` / `"FAIL"`. | | `result_reason` | string | yes | One-line summary. | +| `baseline_check` | object\|null | no | The calibrated golden-limit binning (see below). Non-null only when a `baselines/.json` exists for the target; `null` otherwise. | | `submission_safe` | bool | yes | Orchestrator's assertion that no PII is present. | | `store_location` | string\|null | no | Opt-in only. | | `purchase_date` | string\|null | no | Opt-in only. | @@ -347,6 +348,25 @@ The hash is genuinely useful for **deduplication**: the aggregator can detect re A future hosted aggregator should rotate to HMAC-SHA-256 with a per-deployment secret, so the aggregator-side dedup works but external attackers can't recover the serial. Until that lands, the threat model is: aggregator operator can recover serials; everyone else can't. +## Calibrated baseline check (`baseline_check`) + +Present (non-null) only when `baselines/.json` exists for the `--target` SKU. It is the production-QA-style binning of this unit's metrics against golden control limits derived from known-good units (see [Production QA](../Verification/Production%20QA.md) and `Verification/scripts/make-baseline.sh`). When present, a metric outside its limit escalates `result` to `FAIL`, turning the verdict from advisory (within-unit only) into calibrated. + +```json +"baseline_check": { + "baseline_preset": "mbp-16-m5-max-64", + "n_samples": 12, + "verdict": "pass", + "checks": [ + {"metric": "4_cpu_variance.mean_mb_per_s", "value": 23110.5, "golden": 23089.0, "min": 21934.6, "pass": true}, + {"metric": "12_memory_bandwidth.mean_triad_gb_per_s", "value": 402.1, "golden": 404.0, "min": 383.8, "pass": true}, + {"metric": "5_thermal_load.cpu_die_temp_c.max", "value": 96.4, "golden": 94.8, "max": 101.2, "pass": true} + ] +} +``` + +Each check carries the measured `value`, the `golden` reference, and either a `min` (higher-is-better metrics) or a `max` (lower-is-better), plus `pass`. `verdict` is `fail` if any check fails. `baseline_check` is `null` (and the verdict stays within-unit advisory) when no baseline exists for the SKU, so reports without a baseline omit the binning and remain valid. + ## Versioning policy - **Patch bumps (1.0.0 → 1.0.1)** add optional fields. Old aggregators read the new reports fine. @@ -361,3 +381,4 @@ If a test methodology changes (e.g. variance test gains a new metric, or thresho - **1.1** → added phases 10_race_bench and 11_ssd_test (informational; pass/fail thresholds pending v0.3 calibration corpus). Backward compatible: reports without these phases still validate as 1.0 shape. - **1.2** → added phases 12_memory_bandwidth (informational), 4b_cpu_variance_noaccel (opt-in non-accelerated variance, real pass/warn/fail when run), and 13_gpu_variance (opt-in Metal compute, informational). Added a conservative warn-only floor to 11_ssd_test (`ssd_floor_mb_per_s`) and split the `active-cooled-pro` thermal class into `active-cooled-pro-14` / `active-cooled-pro-16` (the bare class stays valid as the 16"-equivalent, so older reports and presets still compare). Backward compatible: 4b and 13 are `skipped` placeholders unless opted in. - **1.3** → Phase 12 now prefers a vendored STREAM triad (`details.method` is `stream-triad` or the `memmove-proxy` fallback; new `*_triad_gb_per_s` / `mean_scale_gb_per_s` / `mean_add_gb_per_s` fields). Added phase 14_llama_bench (opt-in `--llama`: clones/builds llama.cpp and runs llama-bench, a combined CPU+GPU+memory load; informational, `skipped` unless opted in). Backward compatible: both phase-12 methods keep `mean_copy_gb_per_s`, and 14 is a `skipped` placeholder by default. +- **1.4** → added the optional top-level `baseline_check` object: when `baselines/.json` exists for the target, the run bins each metric against a golden control limit and a metric outside its limit escalates `result` to FAIL (a calibrated verdict instead of within-unit advisory). Added `make-baseline.sh` (builds a baseline from known-good reports) and the `baselines/` directory. Backward compatible: `baseline_check` is `null` and the verdict stays advisory when no baseline exists, so it is not a required field. diff --git a/Verification/Benchmark Reference.md b/Verification/Benchmark Reference.md index b9b9ec7..eff1a3b 100644 --- a/Verification/Benchmark Reference.md +++ b/Verification/Benchmark Reference.md @@ -7,6 +7,8 @@ tags: [verification, benchmarks, reference] Shakedown's own thresholds are not yet calibrated against confirmed-good hardware, and its default workload (SHA-256) is hardware-accelerated, so the harness is a structured *consistency* check, not an absolute pass/fail. Published third-party benchmarks fill the other half: an absolute floor your unit should clear when cool, plus a crowd-sourced distribution of real units you can compare against. This is the "install this, run this, your unit should score about X" companion to the [Runbook](Runbook.md), tuned for catching a bad unit (the M5 Max especially) in a store you cannot easily return to. +> In a hurry at the counter? The [Store Day Checklist](Store%20Day%20Checklist.md) is the short, top-to-bottom run-this-then-that version. This page is the detailed reference behind it (expected scores, sources, caveats). + Use it alongside the harness: - `./run --store` runs the thorough profile (both CPU workloads, GPU, longer warmup, more iterations) for the structured variance and thermal verdict. - `compare-reports.sh REFERENCE.json UNIT.json` diffs your unit against a known-good sibling, which is the most trustworthy signal while the absolute thresholds are uncalibrated. diff --git a/Verification/Production QA.md b/Verification/Production QA.md new file mode 100644 index 0000000..1c20997 --- /dev/null +++ b/Verification/Production QA.md @@ -0,0 +1,51 @@ +--- +created: 2026-06-02 +tags: [verification, methodology, production-qa] +--- + +# Production QA: where Shakedown is, and what factory-grade needs + +Picture a new MacBook Pro coming off the line. Before it ships it should pass a screen good enough to catch the defect class Apple's own QA has been missing on the M5: batch performance variance (a unit that benchmarks fine once, then craters on a repeat run under heat). This doc is the honest engineering gap analysis between what Shakedown is today and what a production-QA station actually requires, plus the architecture to close the distance. + +## What a production-QA screen requires + +Real silicon and laptop production test is not "run a benchmark and eyeball it." It is: + +1. **Calibrated limits, not advisory thresholds.** Pass/fail bands come from characterization of known-good parts, not from public reports. Every limit is defensible against a measured distribution. +2. **A golden reference and binning.** A characterized golden unit (or a population of them) defines the control limits; every unit-under-test is binned against them. Stations are periodically correlated back to the golden so they agree. +3. **Measurement System Analysis (Gage R&R).** Before you trust a pass/fail, you characterize the *measurement system itself*: its repeatability (same unit, same station, repeated) and reproducibility (across stations/conditions). The gage error must be small relative to the tolerance band, or your pass/fail is noise. +4. **Controlled, recorded, gated preconditions.** Ambient temperature, AC power, a quiet system, a defined thermal soak. These are fixtured and held constant, and a run that violates them is rejected, not scored. +5. **Defined defect coverage with a quantified escape rate.** You know which failure modes the screen catches, and the probability a bad unit slips through (the escape rate), backed by seeded-defect or returns data. +6. **Traceability and SPC.** Serial, station, timestamp, and the full measurement record flow into a database. Control charts, Cpk, and yield trends run continuously; drift triggers action. +7. **Takt-appropriate test time.** Functional test is seconds. A *stress screen / burn-in* station is minutes to hours, and that is the right class for a variance defect that only shows under sustained load. + +## Where Shakedown is today + +| Requirement | Status | The gap | +|---|---|---| +| Measurement | Strong | Own workloads (parallel SHA-256, BLAKE2b, STREAM triad, Metal FMA), within-unit variance/decline, sustained thermal stress. Good signal generation. | +| Calibrated limits | Weak | v0.2 thresholds are advisory, derived from public reports, never validated against a confirmed-defective unit. The baseline mechanism (below) now exists, but the limit *data* needs golden samples. | +| Golden reference + binning | Partial | `compare-reports.sh` diffs a unit against one known-good sibling; `make-baseline.sh` + `baselines/` now turn a population of known-good units into limits. Needs the population. | +| MSA / Gage R&R | Absent | We have never characterized our own measurement error (run-to-run on one unit) against the tolerance band. We do not yet know if a "10% spread" limit is signal or gage noise. | +| Controlled conditions | Partial | We record ambient temp and AC state and warn on a non-quiet system, but we do not hard-gate. No fixturing; ambient is whatever the room is. | +| Defect coverage / escape rate | Partial | Catches gross multi-core variance, monotonic decline, dead workers, thermal cliffs. Known holes (from the detection audit): a single slow core averaged across N workers, intermittent defects outside the test window, sub-sample-interval transients, memory/GPU-only faults. Escape rate is unquantified. | +| Traceability / SPC | Partial | Each run emits a SCHEMA-versioned JSON with a hashed serial. The PR-submission flow is the aggregator. No control charts, Cpk, or trend. | +| Takt time | OK as burn-in | The ~45-minute `--store` profile is a stress-screen / burn-in station, not a functional-test station. Appropriate for this defect class. | + +## The architecture to close it + +Built in dependency order, because each leg needs the one before it. + +1. **Calibrated baselines (this wave).** `baselines/.json` holds, per metric, a golden value and a control limit. `make-baseline.sh` computes them from N known-good reports (3-sigma limits once there are enough samples, a tolerance band below that). `./run` loads the baseline for the target SKU when present and bins each metric against it, turning the verdict from "advisory" into a calibrated pass/fail and surfacing it in the verdict banner. When no baseline exists it falls back to today's within-unit-only behavior. This is the golden-reference correlation a factory line is built on. + +2. **Measurement System Analysis (next).** A repeatability harness: run the screen N times back to back on one golden unit, compute the gage CV per metric, and report it against the proposed limit. Until the gage error is small relative to the tolerance, the limits are not trustworthy. This is the step that tells us whether our metrics are even capable of a pass/fail decision. + +3. **Precondition gating.** Promote the soft warnings (not on AC, hot ambient, busy system, chassis not soaked) into hard preconditions that abort or quarantine a run, so every scored run was taken under comparable conditions. Without this, the baseline comparison is apples-to-oranges across rooms. + +4. **Coverage.** Close the audit's holes: statistical single-slow-core attribution (oversubscribe workers, accumulate per-core throughput over a longer run), longer/looped windows for intermittent defects, and treating the memory/GPU phases as real pass/fail once their baselines exist. + +5. **SPC and traceability.** The hosted aggregator with per-SKU control charts, Cpk, and escape-rate tracking. The submission corpus is the data; this is where yield and drift become visible. + +## Honest bottom line + +Shakedown is a strong stress-screen *methodology* with the right instincts. To be a production-QA *station* it needs calibrated limits (data), a characterized gage (MSA), gated conditions, and SPC. This wave builds the calibration mechanism and wires the golden comparison into the one command. The rest is a roadmap, and most of it is gated on one thing the project has always needed and still lacks: a corpus of known-good units to characterize against. The fastest way to start is to run the screen on a unit you trust and `make-baseline.sh` it into the first golden reference. diff --git a/Verification/Store Day Checklist.md b/Verification/Store Day Checklist.md new file mode 100644 index 0000000..8afa4f6 --- /dev/null +++ b/Verification/Store Day Checklist.md @@ -0,0 +1,53 @@ +--- +created: 2026-06-02 +tags: [verification, checklist, store] +--- + +# Store Day Checklist + +You, in the Apple Store, on the charger, verifying the machine you just bought before you walk out. + +Apple's QA catches the obvious build stuff (speakers, keys, screen). What it has been shipping bad on the M5 is **batch performance variance**: a unit that benchmarks fine once and then craters on a repeat run, invisible in a 30-second look. That is the whole point of this tool, and it is what one command checks. + +## Setup (2 min) + +- Plug in the charger. +- Get to the desktop (skip Apple ID with "Set Up Later"). **Set a login password** (one step needs it). Join the store Wi-Fi. + +## The one command + +```bash +xcode-select --install +git clone https://github.com/ugglr/mac-shakedown ~/mac-shakedown +cd ~/mac-shakedown +./run --target mbp-16-m5-max-64 --store +``` + +- `xcode-select --install` is a one-time GUI dialog (~10 min); the rest is one paste. +- Swap `mbp-16-m5-max-64` for your SKU (see `targets/`), or drop `--target` to auto-detect. +- Enter your login password when it asks for sudo (the thermal phase needs it). +- `--store` is the thorough profile: ~45 min of CPU variance (accelerated **and** non-accelerated), a 10-minute sustained thermal load, the memory triad, and the GPU pass. It takes over; walk around while it runs. +- Want the exact workload the M5 defect was reported on? Add `--llama` (it builds llama.cpp and runs an LLM load; extra time). + +## Read the verdict + +It ends with a banner: **`SHAKEDOWN VERDICT: PASS`** or **`FAIL`**. + +- **PASS** = no performance-defect signature across variance, thermal, memory, GPU. +- A single **WARN** = rerun that phase once (`./Verification/scripts/cpu-variance.sh`). Clears, fine. Warns again, treat as fail. +- **FAIL**, or a WARN that repeats = **exchange at the counter, now.** + +## Turn "advisory" into "confident" + +The thresholds are still v0.2 (derived from public reports, not yet calibrated against confirmed-good silicon). The one thing that makes a PASS trustworthy is a **comparison**, and it is performance, not cosmetics: + +- Run the same `./run --store` (or just Geekbench 6 once + a Cinebench 2024 loop) on a **floor demo of the same model**, and confirm yours lands within a few percent. +- Or diff against a known-good report for your SKU: + ```bash + ./Verification/scripts/compare-reports.sh known-good.json Reports/submissions/your-report.json + ``` +- Or eyeball your Geekbench / Cinebench numbers against the published baseline in [Benchmark Reference](Benchmark%20Reference.md) (or live at mianibench.com). A unit 15-20%+ below baseline, or one that swings more than ~10% run to run, is the signature. + +## If it fails + +Exchange at the counter before you leave the store. Apple Hong Kong is exchange-only and only for a verified defect, so catching it in-store is by far the easiest path. Keep the receipt and the box until it passes. diff --git a/Verification/scripts/make-baseline.sh b/Verification/scripts/make-baseline.sh new file mode 100755 index 0000000..35a18b9 --- /dev/null +++ b/Verification/scripts/make-baseline.sh @@ -0,0 +1,121 @@ +#!/bin/bash +# make-baseline.sh: build a calibrated baseline from known-good Shakedown reports. +# +# A production screen needs limits derived from known-good parts, not from public +# reports. This ingests N reports from KNOWN-GOOD units of the SAME SKU and, per +# metric, computes a golden value (the median) and an acceptance limit: +# - with >= 8 samples: a 3-sigma control limit (golden -+ 3*stdev) +# - below that: a tolerance band (golden * (1 -+ BASELINE_TOLERANCE_PCT)) +# `./run` bins a unit-under-test against baselines/.json when it exists, +# turning the verdict from advisory into a calibrated pass/fail. Regenerate as the +# corpus grows; more known-good samples means tighter, more trustworthy limits. +# +# Usage: +# ./Verification/scripts/make-baseline.sh good1.json good2.json ... > baselines/.json +# +# Env: +# BASELINE_TOLERANCE_PCT default 15 (band used when fewer than 8 samples) + +set -euo pipefail + +if [[ $# -lt 1 ]]; then + echo "usage: $(basename "$0") good1.json [good2.json ...] > baselines/.json" >&2 + echo " inputs are Shakedown reports from KNOWN-GOOD units of the same SKU" >&2 + exit 2 +fi + +for f in "$@"; do + if [[ ! -f "$f" ]]; then + echo "make-baseline.sh: file not found: $f" >&2 + exit 2 + fi +done + +TOL=${BASELINE_TOLERANCE_PCT:-15} +if ! [[ "$TOL" =~ ^[0-9]+$ ]]; then + echo "make-baseline.sh: BASELINE_TOLERANCE_PCT must be an integer (got '$TOL')" >&2 + exit 2 +fi + +python3 - "$TOL" "$@" <<'PYEOF' +import json +import statistics +import sys + +tol = float(sys.argv[1]) / 100.0 +paths = sys.argv[2:] + +# (phase_key, details-path, higher_is_better). These are the metrics a baseline +# bins against. Higher-is-better gets a `min` limit; lower-is-better a `max`. +METRICS = [ + ("4_cpu_variance", ["mean_mb_per_s"], True), + ("4_cpu_variance", ["spread_pct"], False), + ("4b_cpu_variance_noaccel", ["mean_mb_per_s"], True), + ("12_memory_bandwidth", ["mean_triad_gb_per_s"], True), + ("12_memory_bandwidth", ["mean_copy_gb_per_s"], True), + ("13_gpu_variance", ["mean_gflops"], True), + ("5_thermal_load", ["steady_state_vs_peak_pct"], True), + ("5_thermal_load", ["cpu_die_temp_c", "max"], False), + ("10_race_bench", ["throughput_mb_per_s"], True), + ("11_ssd_test", ["read_mb_per_s"], True), + ("11_ssd_test", ["write_mb_per_s"], True), +] + +reports = [] +for p in paths: + with open(p) as f: + reports.append(json.load(f)) + + +def get(report, phase_key, path): + node = report.get("phases", {}).get(phase_key, {}).get("details", {}) + for key in path: + if not isinstance(node, dict): + return None + node = node.get(key) + return node if isinstance(node, (int, float)) else None + + +chips = {r.get("unit", {}).get("chip") for r in reports} +mems = {r.get("unit", {}).get("memory_gb") for r in reports} +if len([c for c in chips if c]) > 1 or len([m for m in mems if m]) > 1: + print("make-baseline.sh: WARNING: inputs span more than one SKU " + f"(chips={sorted(c for c in chips if c)}, memory={sorted(m for m in mems if m)}). " + "A baseline is only meaningful within one chip + memory config.", file=sys.stderr) + +preset = reports[0].get("target", {}).get("preset") +metrics = {} +for phase_key, path, higher in METRICS: + vals = [v for v in (get(r, phase_key, path) for r in reports) if v is not None] + if not vals: + continue + golden = round(statistics.median(vals), 3) + entry = {"golden": golden, "higher_better": higher, "n": len(vals)} + if len(vals) >= 8: + sd = statistics.stdev(vals) + entry["sigma"] = round(sd, 3) + if higher: + entry["min"] = round(golden - 3 * sd, 3) + else: + entry["max"] = round(golden + 3 * sd, 3) + else: + if higher: + entry["min"] = round(golden * (1 - tol), 3) + else: + entry["max"] = round(golden * (1 + tol), 3) + metrics[phase_key + "." + ".".join(path)] = entry + +out = { + "preset": preset, + "chip": (sorted(c for c in chips if c)[0] if len([c for c in chips if c]) == 1 + else sorted(c for c in chips if c)), + "memory_gb": (sorted(m for m in mems if m)[0] if len([m for m in mems if m]) == 1 + else sorted(m for m in mems if m)), + "n_samples": len(reports), + "tolerance_pct": float(sys.argv[1]), + "note": ("golden = median of known-good units; limit = 3-sigma control limit " + "when n >= 8, else golden +- tolerance. Regenerate as the corpus grows."), + "metrics": metrics, +} +print(json.dumps(out, indent=2)) +PYEOF diff --git a/Verification/scripts/run-shakedown.sh b/Verification/scripts/run-shakedown.sh index 22db4fe..4ec2082 100755 --- a/Verification/scripts/run-shakedown.sh +++ b/Verification/scripts/run-shakedown.sh @@ -446,7 +446,7 @@ import sys local_dir, submissions_dir, target_name, notes) = sys.argv[1:17] SHAKEDOWN_VERSION = "0.1.0" -SCHEMA_VERSION = "1.3" +SCHEMA_VERSION = "1.4" def load(path): with open(path) as f: @@ -723,6 +723,47 @@ else: result = "PASS" result_reason = "all phases passed; no defect signatures detected" +# Calibrated baseline check (production-QA style). When baselines/.json +# exists for this target, bin each metric against its golden control limit; a +# metric outside its limit escalates the overall result to FAIL. Absent a +# baseline the verdict stays within-unit-only (advisory). See Production QA.md. +baseline_check = None +if target_name: + repo_root = os.path.dirname(os.path.dirname(local_dir)) + baseline_path = os.path.join(repo_root, "baselines", target_name + ".json") + if os.path.exists(baseline_path): + with open(baseline_path) as bf: + baseline = json.load(bf) + checks = [] + for key, spec in (baseline.get("metrics") or {}).items(): + pk, _, mpath = key.partition(".") + node = phases.get(pk, {}).get("details", {}) + for part in mpath.split("."): + node = node.get(part) if isinstance(node, dict) else None + if not isinstance(node, (int, float)): + continue + higher = spec.get("higher_better", True) + entry = {"metric": key, "value": node, "golden": spec.get("golden"), "pass": True} + if higher and spec.get("min") is not None: + entry["min"] = spec["min"] + entry["pass"] = node >= spec["min"] + elif (not higher) and spec.get("max") is not None: + entry["max"] = spec["max"] + entry["pass"] = node <= spec["max"] + checks.append(entry) + failed = [c for c in checks if not c["pass"]] + baseline_check = { + "baseline_preset": baseline.get("preset"), + "n_samples": baseline.get("n_samples"), + "checks": checks, + "verdict": "fail" if failed else "pass", + } + if failed: + reason = "below golden baseline: " + ", ".join( + f"{c['metric']} {c['value']} vs golden {c.get('golden')}" for c in failed) + result_reason = (result_reason + "; " + reason) if result == "FAIL" else reason + result = "FAIL" + if target: target_block = {"preset": target_name} target_block.update({k: v for k, v in target.items() if not k.startswith("_")}) @@ -742,6 +783,7 @@ report_full = { "phases": phases, "result": result, "result_reason": result_reason, + "baseline_check": baseline_check, "submission_safe": True, "store_location": None, "purchase_date": None, @@ -793,17 +835,36 @@ print(json.dumps({ "submission_path": submission_path, "submission_safe": report_sub["submission_safe"], }, indent=2)) -PYEOF - -cat <&2 -shakedown: done. - -Next steps: - 1. Review the submission JSON for any leftover PII: - cat Reports/submissions/ - 2. If you ran the manual phases (display / physical / Apple Diagnostics), - hand-edit the local file then copy a sanitized version to submissions/. - 3. Open a PR adding the submission JSON to help calibrate v0.1 thresholds. - See CONTRIBUTING.md for the submission flow. -INFO +# Human-readable verdict banner to stderr, printed last so it is the final +# thing on screen: the "by the end I know" payoff. +_tty = sys.stderr.isatty() +_c = ("\033[1;32m" if result == "PASS" else "\033[1;31m") if _tty else "" +_d = "\033[2m" if _tty else "" +_x = "\033[0m" if _tty else "" +_bar = "=" * 66 +print(f"\n{_c}{_bar}", file=sys.stderr) +print(f" SHAKEDOWN VERDICT: {result}", file=sys.stderr) +print(f"{_bar}{_x}", file=sys.stderr) +print(f" {result_reason}", file=sys.stderr) +if baseline_check is not None: + _ck = baseline_check.get("checks") or [] + _within = sum(1 for c in _ck if c.get("pass")) + print(f"{_d} Binned against the golden baseline for {baseline_check.get('baseline_preset')} " + f"({baseline_check.get('n_samples')} known-good samples):{_x}", file=sys.stderr) + print(f"{_d} {_within}/{len(_ck)} metrics within their golden limits. This is a calibrated" + f" verdict.{_x}", file=sys.stderr) +else: + if result == "PASS": + print(f"{_d} No batch performance-defect signature: CPU variance, thermal, memory,{_x}", + file=sys.stderr) + print(f"{_d} and GPU all within range (the defect class Apple QA has been missing).{_x}", + file=sys.stderr) + print(f"{_d} No golden baseline for this SKU yet, so this verdict is within-unit only", + file=sys.stderr) + print(f" (advisory). For a calibrated verdict, build one from known-good units with", + file=sys.stderr) + print(f" make-baseline.sh, or compare against one with compare-reports.sh.{_x}", file=sys.stderr) +print(f"{_d} Report saved: {submission_path}{_x}", file=sys.stderr) +print(f"{_c}{_bar}{_x}\n", file=sys.stderr) +PYEOF diff --git a/baselines/README.md b/baselines/README.md new file mode 100644 index 0000000..e371938 --- /dev/null +++ b/baselines/README.md @@ -0,0 +1,11 @@ +# Baselines + +Calibrated golden baselines, one `.json` per `targets/.json`. Each holds, per metric, a golden value and a control limit, generated from reports of KNOWN-GOOD units of that SKU with [`make-baseline.sh`](../Verification/scripts/make-baseline.sh): + +```bash +./Verification/scripts/make-baseline.sh good1.json good2.json ... > baselines/mbp-16-m5-max-64.json +``` + +When `baselines/.json` exists, `./run --target ` bins each measured metric against its golden control limit and the verdict becomes a **calibrated** pass/fail: a metric outside its limit fails the run, and the verdict banner says so. Absent a baseline, the verdict is within-unit only (advisory). + +This directory ships mostly empty on purpose. A baseline is only as good as the known-good units it was built from, and that corpus is still being gathered. See [`Verification/Production QA.md`](../Verification/Production%20QA.md) for the calibration discipline (golden reference, 3-sigma control limits, the path to production-grade). Build your own from a unit you trust to get a calibrated verdict today.