diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 458a755..b306bda 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -72,7 +72,7 @@ jobs: "5_thermal_load", "6_display", "7_physical", "8_apple_diagnostics", "9_idle_drain", "10_race_bench", "11_ssd_test", "12_memory_bandwidth", - "13_gpu_variance"] + "13_gpu_variance", "14_llama_bench"] FILENAME_RE = re.compile(r"^\d{4}-\d{2}-\d{2}-[a-z0-9-]+-[0-9a-f]{4}\.json$") def find_keys(obj, predicate, path=""): diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ef8eef..ccf71f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ All notable changes to this project are documented here. Format follows [Keep a ## [Unreleased] +### Added: in-store verification toolkit (wave 8) + +- **`--store` thorough profile.** One flag for verifying a new unit (M5 especially): turns on `--noaccel` and `--gpu` and runs a longer warmup with more iterations, so an intermittent batch defect has more chances to surface. Honors any timing env vars the user set. +- **`compare-reports.sh`.** Diffs two reports side by side (a known-good sibling vs the unit under test) and flags every metric where the unit materially trails. Since the absolute thresholds aren't calibrated yet, this unit-to-unit comparison is the most trustworthy in-store signal. +- **Benchmark Reference** (`Verification/Benchmark Reference.md`). Install-this / run-this / expected-score guide with sourced, adversarially-verified Cinebench R24 + Geekbench 6 baselines per generation (M1-M5 families + Intel 2019), the in-store and hotel protocols, the Hong Kong return constraint, and live-lookup links (Geekbench Browser, mianibench.com crowd distributions). Corrected the stale M5 Max baseline in the m5-2026 calibration (~25-28k to ~29,000-29,400 GB6 multi-core). +- **Phase 12 STREAM triad.** Memory bandwidth now prefers a vendored single-file STREAM triad (`stream-triad.c`: real copy / scale / add / triad), compiled at runtime with clang, no network. Falls back to the pure-Python `memmove` proxy when clang is unavailable. Closes the copy-only-proxy gap; `details.method` records which ran. +- **Phase 14 `--llama` (opt-in).** Clones and builds llama.cpp at a pinned ref and runs `llama-bench`, a combined CPU+GPU+memory AI load (the workload class the M5 defect was reported on). Off by default; it reaches the network and runs third-party code (disclosed in SECURITY.md) and skips cleanly without git / cmake / network / model. +- **Schema 1.3.** Phase 12 gains the triad method and `*_triad_gb_per_s` fields; new phase key `14_llama_bench` (a `skipped` placeholder by default), added to the CI submission-audit required-phase list. Backward compatible: both phase-12 methods keep `mean_copy_gb_per_s`, and 14 is skipped unless opted in. + ### Added: roadmap completion (wave 7) - **Phase 4b non-accelerated CPU variance** (`cpu-variance.sh WORKLOAD=blake2b`, opt-in `./run --noaccel`). Reruns the Phase 4 variance methodology with BLAKE2b, which has no dedicated CPU instruction and so exercises the integer pipelines and memory instead of the SHA engine. Catches batch defects that SHA-NI / the Apple crypto coprocessor hide. Same script and verdict logic as Phase 4 (no duplicated thresholds): real `pass`/`warn`/`fail` when run, `skipped` placeholder otherwise. Runs on the already-hot chassis after Phase 4 with a short re-warm and no cold burst. Adds ~6 min when enabled. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index df4f8f9..d3b6ff6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -52,7 +52,7 @@ The verification scripts and runbook **don't need to change**. They're calibrati ## Improving a script -The scripts in `Verification/scripts/` are deliberately self-contained: bash plus Python heredocs, no external dependencies beyond what ships with macOS. Keep it that way. If you need to add a third-party benchmark (Geekbench, sysbench), or something that genuinely needs more than the stdlib (the opt-in `--gpu` phase compiles a Metal kernel with `swiftc` at runtime), make it optional and have it skip cleanly when the dependency is absent, so the default `./run` stays pure bash plus Python stdlib. +The scripts in `Verification/scripts/` are deliberately self-contained: bash plus Python heredocs, no external dependencies beyond what ships with macOS. Keep it that way. Two patterns in the tree show how to go further without breaking that: a vendored single-file source compiled at runtime with a pure fallback (the Phase 12 STREAM triad, `stream-triad.c`, falls back to a `memmove` proxy, and never touches the network), and a strictly opt-in phase that may use the network or third-party code (the `--gpu` Metal compile and the `--llama` llama.cpp clone/build), off by default, skipping cleanly when unavailable, and disclosed in [SECURITY.md](SECURITY.md). The default `./run` stays pure bash plus Python stdlib and offline; anything beyond that is opt-in and disclosed. ### Validating changes to a script diff --git a/README.md b/README.md index 29b161f..28a3619 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,9 @@ Or without a preset, which auto-detects chassis class from `system_profiler` (fi ./run ``` -The orchestrator runs the automated phases (preflight → inventory → battery → race benchmark → SSD test → memory bandwidth → CPU variance → thermal load) end-to-end, asks for sudo once upfront (Phase 5 and the SSD page-cache drop need it), and writes a SCHEMA-compliant report to `Reports/local/` plus a sanitized PR-able copy to `Reports/submissions/`. Two heavier passes are opt-in: `--noaccel` adds a non-accelerated (BLAKE2b) variance pass, and `--gpu` adds a Metal GPU compute pass. Runtime ~20 min on Intel, ~27 min on Air, ~47 min on MacBook Pro (add ~6 min for `--noaccel`). +The orchestrator runs the automated phases (preflight → inventory → battery → race benchmark → SSD test → memory bandwidth → CPU variance → thermal load) end-to-end, asks for sudo once upfront (Phase 5 and the SSD page-cache drop need it), and writes a SCHEMA-compliant report to `Reports/local/` plus a sanitized PR-able copy to `Reports/submissions/`. Opt-in flags add heavier passes: `--noaccel` (a non-accelerated BLAKE2b variance pass), `--gpu` (a Metal GPU compute pass), and `--llama` (clones and builds llama.cpp for a combined CPU+GPU+memory AI load). `--store` bundles the thorough profile for verifying a new unit. Runtime ~20 min on Intel, ~27 min on Air, ~47 min on MacBook Pro. + +> Verifying a new purchase, especially in a store you can't easily return to? The [Benchmark Reference](Verification/Benchmark%20Reference.md) is the install-this / run-this / expected-score guide (sourced per-generation Cinebench + Geekbench baselines, the in-store and hotel protocols, and crowd-sourced live-lookup links). To diff your unit against a known-good sibling, run `Verification/scripts/compare-reports.sh reference.json yours.json`. `./run --no-sudo` skips the 10-min thermal phase (the only phase that needs sudo) for a half-runtime no-password variance-only pass. @@ -61,11 +63,12 @@ See [`examples/sample-report-illustrative/`](examples/sample-report-illustrative | 3 | Sensor & port inventory | (uses Phase 1 output) | | 10 | Race benchmark (cold) | `xz -9 -T

` of a 200 MB random blob → `race-bench.sh` | | 11 | SSD sequential read/write (cold) | 2 GB incompressible random data, page cache dropped between → `ssd-test.sh` | -| 12 | Memory bandwidth (cold) | multi-threaded `memmove` over cache-busting buffers → `memory-bandwidth.sh` | +| 12 | Memory bandwidth (cold) | multi-threaded STREAM triad (vendored C, clang at runtime) with a `memmove` fallback → `memory-bandwidth.sh` | | 4 | **CPU performance variance** | 5 s burst + chassis-class-aware warmup (300 s on Pro, 60 s on Air) + 5 × 60 s timed iterations, parallel SHA-256 → `cpu-variance.sh` | | 4b | Non-accelerated CPU variance (opt-in `--noaccel`) | same methodology with BLAKE2b (no crypto instruction, hits the integer pipelines) → `cpu-variance.sh` | | 5 | Sustained thermal load (chassis-class-aware thresholds) | 10 min continuous + `powermetrics` sampling → `thermal-load.sh` | | 13 | GPU compute variance (opt-in `--gpu`) | sustained Metal FMA load, compiled with `swiftc` at runtime → `gpu-variance.sh` | +| 14 | Combined CPU+GPU+memory (opt-in `--llama`) | clones + builds llama.cpp, runs `llama-bench` (AI inference load) → `llama-bench.sh` | | 6 | Display visual inspection | fullscreen color cycle in Safari → `display-test.sh` | | 7 | Manual physical inspection | hinge, keyboard, speakers, ports, Touch ID, etc. (runbook checklist) | | 8 | Apple Diagnostics | reboot + Cmd-D | @@ -114,16 +117,21 @@ mac-shakedown/ ├── Verification/ # generation-agnostic test machinery │ ├── Runbook.md # the procedure │ ├── Pass-Fail Criteria.md # thresholds (parameterized by chassis class) +│ ├── Benchmark Reference.md # install/run/expected-score + in-store protocol +│ ├── per-core-pinning.md # why macOS has no per-core affinity (methodology note) │ └── scripts/ │ ├── run-shakedown.sh # the orchestrator (`./run` execs this) │ ├── inventory.sh # system_profiler + sysctl → JSON │ ├── battery.sh # ioreg battery health → JSON │ ├── race-bench.sh # cold xz race → JSON │ ├── ssd-test.sh # sequential SSD read/write → JSON (sudo for purge) -│ ├── memory-bandwidth.sh # multi-threaded memmove bandwidth → JSON +│ ├── memory-bandwidth.sh # STREAM triad (stream-triad.c) or memmove fallback → JSON +│ ├── stream-triad.c # vendored STREAM-style triad, compiled at runtime │ ├── cpu-variance.sh # burst + warmup + 5×60s timed iters → JSON (WORKLOAD=blake2b for 4b) │ ├── thermal-load.sh # 10-min sustained + powermetrics → JSON (sudo) │ ├── gpu-variance.sh # opt-in Metal compute variance, swiftc at runtime → JSON +│ ├── llama-bench.sh # opt-in llama.cpp combined load (clone+build) → JSON +│ ├── compare-reports.sh # diff two reports (unit vs known-good sibling) │ └── display-test.sh # fullscreen color cycle (HTML) ├── targets/ # preset SKU configs │ ├── README.md @@ -180,12 +188,13 @@ The manual phases (display, physical inspection, Apple Diagnostics, idle drain) Wave 7 shipped most of the original roadmap: the non-accelerated variance pass (Phase 4b, BLAKE2b), GPU compute variance (Phase 13, Metal), memory bandwidth (Phase 12), a conservative SSD floor, the `active-cooled-pro-14` / `active-cooled-pro-16` thermal sub-classes, and M1-M4 plus Intel-era generation calibrations. +Wave 8 added the in-store toolkit: the `--store` thorough profile, `compare-reports.sh` (diff your unit against a known-good sibling), a sourced [Benchmark Reference](Verification/Benchmark%20Reference.md), a vendored STREAM triad in Phase 12 (real copy / scale / add / triad), and an opt-in `--llama` phase that builds llama.cpp for a combined CPU+GPU+memory load. + Still open: - **Hosted aggregator.** Eventually, submission via API to a public site so reports aren't reviewed by hand. Until then, the PR-submission flow above *is* the aggregator. Slower, but no infra, and PR review catches PII before merge. - **Calibrated v0.3 thresholds.** The race / SSD / memory-bandwidth / GPU phases are informational in v0.2. Once the submission corpus has per-SKU baselines, they get chassis-family pass/fail bands. The SSD floor is the only band today, and it's a conservative sanity floor (flags below 500 MB/s with the cache dropped), not a calibrated range. - **Per-core pinning.** macOS lacks public CPU affinity APIs, so we can't pin workers to specific cores; a defective single core gets averaged across N P-cores. Reporting `worker_imbalance_pct_per_iter` is the partial mitigation. The investigation is written up in [`Verification/per-core-pinning.md`](Verification/per-core-pinning.md): the short version is that `THREAD_AFFINITY_POLICY` is ignored on Apple Silicon and QoS hints only steer between the P and E clusters, so there's no per-core pinning to be had today. -- **Full STREAM triad.** Phase 12 measures copy bandwidth via `memmove` (a lower bound on the memory subsystem). Scale / add / triad need a native kernel, which would break the pure-stdlib property of the default run. ## Origin diff --git a/Reports/SCHEMA.md b/Reports/SCHEMA.md index 4a57f52..405e0d4 100644 --- a/Reports/SCHEMA.md +++ b/Reports/SCHEMA.md @@ -9,7 +9,7 @@ Every QA run produces two artifacts in `Reports/`: | Field | Type | Required | Notes | |---|---|---|---| -| `schema_version` | string | yes | Semver of the schema. Currently `"1.2"`. | +| `schema_version` | string | yes | Semver of the schema. Currently `"1.3"`. | | `shakedown_version` | string | yes | Tool version that produced the report (e.g. `"0.1.0"`). | | `timestamp` | string | yes | ISO 8601 UTC. | | `illustrative` | bool | no | `true` for hand-crafted samples (under `examples/sample-report-*`). Aggregator must filter these out. | @@ -205,39 +205,47 @@ As of schema 1.2 the phase carries a conservative warn-only floor (`ssd_floor_mb ## Phase 12 (`12_memory_bandwidth`): full detail -Produced by `memory-bandwidth.sh`. Multi-threaded sequential `ctypes.memmove` copy, one worker per P-core, over buffers sized to exceed the system-level cache so it measures DRAM bandwidth, not cache. Comparable across submissions grouped by `unit.chip` + `unit.memory_gb`. +Produced by `memory-bandwidth.sh`. Measures DRAM bandwidth with one worker thread per P-core over buffers sized past the system-level cache. A single thread can't saturate the multi-channel controllers on Apple Silicon, so it is multi-threaded the way STREAM uses OpenMP. Comparable across submissions grouped by `unit.chip` + `unit.memory_gb`. + +Two methods, recorded in `details.method`: +- **`stream-triad`** (preferred): the vendored `stream-triad.c` (real STREAM Copy / Scale / Add / Triad), compiled at runtime with the `clang` from the Command Line Tools. No network, no shipped binary; the C source is in the repo, compiled to a tempfile and deleted. +- **`memmove-proxy`** (fallback): a pure-Python `ctypes.memmove` copy loop, used when `clang` is unavailable. A lower bound on the triad (copy only), but it still catches a slow or inconsistent memory subsystem. ```json { "verdict": "info", - "duration_s": 10, + "duration_s": 6, "details": { - "workload": "multi-threaded sequential memcpy (ctypes.memmove), DRAM-bound", + "workload": "STREAM-style triad (vendored C, pthreads), DRAM-bound", + "method": "stream-triad", "workers": 12, - "per_buffer_mb": 64, - "total_working_set_mb": 1536, - "iterations": 5, - "seconds_per_iter": 2, - "copy_bandwidth_gb_per_s": [98.2, 97.8, 98.5, 98.1, 97.9], - "mean_copy_gb_per_s": 98.1, - "min_copy_gb_per_s": 97.8, - "max_copy_gb_per_s": 98.5, - "spread_pct": 0.71, - "touched_bandwidth_gb_per_s_mean": 196.2, - "wall_seconds": 10.4, + "working_set_mb": 1023, + "reps": 5, + "copy_gb_per_s": [412.1, 410.0, 408.6, 409.9, 410.3], + "scale_gb_per_s": [389.0, 387.2, 388.4, 388.1, 387.8], + "add_gb_per_s": [403.1, 402.0, 401.8, 402.9, 402.7], + "triad_gb_per_s": [404.5, 403.1, 401.1, 406.3, 404.0], + "mean_copy_gb_per_s": 410.18, + "mean_scale_gb_per_s": 388.1, + "mean_add_gb_per_s": 402.5, + "mean_triad_gb_per_s": 403.8, + "min_triad_gb_per_s": 401.1, + "max_triad_gb_per_s": 406.3, + "spread_pct": 1.29, + "wall_seconds": 6, "data_quality": "ok", "data_quality_notes": [] }, "verdict_reasons": [ - "mean copy 98.1 GB/s (read+write ~196.2 GB/s touched) across 12 workers, spread 0.7%", + "triad mean 403.8 GB/s (copy 410.2, scale 388.1, add 402.5) across 12 threads, spread 1.3%", "informational in v0.2; calibrated pass/fail thresholds land in v0.3" ] } ``` -`copy_bandwidth_gb_per_s` counts bytes copied (N per `memmove`). Each copy also reads N and writes N, so `touched_bandwidth_gb_per_s_mean` (2x the copy mean) is the read+write traffic the controller actually moved. This is a lower bound on a full STREAM triad: scale / add / triad need elementwise arithmetic that pure Python can't do at memory speed without a native kernel. `memmove` is the honest stdlib proxy, and it still catches a memory subsystem that is slow or inconsistent across runs. A single thread cannot saturate the multi-channel controllers on Apple Silicon, so the test is multi-threaded the way STREAM is built with OpenMP. +The `memmove-proxy` fallback emits the older copy-only shape instead (`copy_bandwidth_gb_per_s`, `mean_copy_gb_per_s`, `touched_bandwidth_gb_per_s_mean`, `per_buffer_mb`, `iterations`, `seconds_per_iter`) with `method: "memmove-proxy"` and a `data_quality_notes` entry explaining the triad was unavailable. Both methods emit `mean_copy_gb_per_s` so reports stay comparable. -`verdict: "skipped"` when there is not enough RAM to size cache-busting buffers for the worker count (the working set is capped at 40% of physical memory). +`verdict: "skipped"` when there is not enough RAM to size cache-busting buffers (the working set is capped at 40% of physical memory). ## Phase 4b (`4b_cpu_variance_noaccel`): full detail @@ -285,9 +293,43 @@ This is the one phase that is not pure bash + Python stdlib: a real GPU load nee `verdict: "skipped"` when `--gpu` is not passed, or when the GPU / toolchain is unavailable. +## Phase 14 (`14_llama_bench`): full detail + +Produced by `llama-bench.sh`. Opt-in (`./run --llama`). The only phase that reaches the network and runs third-party code: it clones and builds llama.cpp (pinned ref, cached) with `cmake`, then runs `llama-bench` on a small GGUF model and records prompt and generation tokens-per-second plus run-to-run spread. An LLM inference load is the one test that stresses the CPU, the GPU (Metal), and the memory subsystem together under sustained load, which is the workload class the M5 Max defect was reported on. + +```json +{ + "verdict": "info", + "duration_s": 90, + "details": { + "workload": "llama.cpp llama-bench (combined CPU + GPU + memory, AI inference)", + "model": "qwen2.5-0.5b-instruct-q4_k_m.gguf", + "llama_ref": "b4585", + "backend": "Metal", + "prompt_tok_per_s": 4200.5, + "prompt_tok_per_s_stddev": 18.3, + "prompt_spread_pct": 0.44, + "gen_tok_per_s": 180.7, + "gen_tok_per_s_stddev": 1.2, + "gen_spread_pct": 0.66, + "wall_seconds": 90, + "data_quality": "ok", + "data_quality_notes": [] + }, + "verdict_reasons": [ + "Metal: prompt 4,200.5 tok/s, gen 180.7 tok/s (gen spread 0.7%)", + "informational in v0.2; GPU/AI thresholds land once the corpus has baselines" + ] +} +``` + +This phase is neither pure stdlib nor offline: opting in clones a third-party repo, builds it, may download a model, and runs it. The model comes from `LLAMA_MODEL` (a local path) or a pinned `LLAMA_MODEL_URL` download. It degrades to `verdict: "skipped"` (never failing the run) when `git`, `cmake`, the network, or a model is unavailable. See [SECURITY.md](../SECURITY.md) for the disclosure. + +`verdict: "skipped"` when `--llama` is not passed, or when any of git / cmake / network / model is unavailable. + ## Note on phase key ordering -The phase keys (`0_preflight`, `1_inventory`, ... `4b_cpu_variance_noaccel`, `10_race_bench`, `12_memory_bandwidth`, `13_gpu_variance`) sort alphabetically rather than numerically (`10` comes before `2` lexicographically, and `4b` sorts after `4`). JSON output preserves the orchestrator's insertion order so human-readable reports render in run order: preflight, inventory, battery, sensors, race, ssd, memory_bandwidth, cpu_variance, cpu_variance_noaccel, thermal_load, gpu_variance, then the skipped manual phases. Parsers iterating phase names should not assume numeric sort order. +The phase keys (`0_preflight`, `1_inventory`, ... `4b_cpu_variance_noaccel`, `10_race_bench`, `12_memory_bandwidth`, `13_gpu_variance`, `14_llama_bench`) sort alphabetically rather than numerically (`10` comes before `2` lexicographically, and `4b` sorts after `4`). JSON output preserves the orchestrator's insertion order so human-readable reports render in run order: preflight, inventory, battery, sensors, race, ssd, memory_bandwidth, cpu_variance, cpu_variance_noaccel, thermal_load, gpu_variance, llama_bench, then the skipped manual phases. Parsers iterating phase names should not assume numeric sort order. ## Privacy / submission-safety @@ -318,3 +360,4 @@ If a test methodology changes (e.g. variance test gains a new metric, or thresho - **1.0** → initial release schema (phases 0-9). - **1.1** → added phases 10_race_bench and 11_ssd_test (informational; pass/fail thresholds pending v0.3 calibration corpus). Backward compatible: reports without these phases still validate as 1.0 shape. - **1.2** → added phases 12_memory_bandwidth (informational), 4b_cpu_variance_noaccel (opt-in non-accelerated variance, real pass/warn/fail when run), and 13_gpu_variance (opt-in Metal compute, informational). Added a conservative warn-only floor to 11_ssd_test (`ssd_floor_mb_per_s`) and split the `active-cooled-pro` thermal class into `active-cooled-pro-14` / `active-cooled-pro-16` (the bare class stays valid as the 16"-equivalent, so older reports and presets still compare). Backward compatible: 4b and 13 are `skipped` placeholders unless opted in. +- **1.3** → Phase 12 now prefers a vendored STREAM triad (`details.method` is `stream-triad` or the `memmove-proxy` fallback; new `*_triad_gb_per_s` / `mean_scale_gb_per_s` / `mean_add_gb_per_s` fields). Added phase 14_llama_bench (opt-in `--llama`: clones/builds llama.cpp and runs llama-bench, a combined CPU+GPU+memory load; informational, `skipped` unless opted in). Backward compatible: both phase-12 methods keep `mean_copy_gb_per_s`, and 14 is a `skipped` placeholder by default. diff --git a/SECURITY.md b/SECURITY.md index e2d2e56..42d4c1a 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,6 +1,6 @@ # Security -Shakedown runs entirely on the user's own Mac. The surface area is small: a handful of short shell scripts plus inline Python heredocs, no external dependencies beyond what ships with macOS, no outbound network calls, and no daemons or persistent components. Everything is local. Auditing the scripts before running them is encouraged. They are pure bash and Python, no minified or obfuscated code, and short enough to read end to end in a few minutes. One exception, opt-in only: the `--gpu` phase compiles a small Metal kernel at runtime (see below). +Shakedown runs entirely on the user's own Mac. The surface area is small: a handful of short shell scripts plus inline Python heredocs, no external dependencies beyond what ships with macOS, and no daemons or persistent components. The default `./run` makes no outbound network calls and runs no third-party code. Everything is local and auditable: the scripts are pure bash and Python, no minified or obfuscated code, short enough to read end to end in a few minutes. Two opt-in exceptions, both off by default and disclosed below: the `--gpu` phase compiles a small Metal kernel at runtime, and the `--llama` phase clones and builds llama.cpp and reaches the network. ## What the scripts do with elevated privileges @@ -8,11 +8,15 @@ Shakedown runs entirely on the user's own Mac. The surface area is small: a hand - No other script asks for, requires, or uses elevated privileges. - All other data collection is unprivileged: SHA-256 over a fixed in-memory 1 MB buffer for the CPU workload (deterministic; no I/O), hardware facts via `system_profiler` / `ioreg` / `sysctl`, and battery state via `ioreg AppleSmartBattery`. - Writes are confined to the repo's `Reports/` directory and short-lived tempfiles under `/var/folders/.../shakedown-*` (the system per-user temp area). Nothing is written elsewhere. -- No outbound network requests are made by any script. Reports stay on the local disk unless the user chooses to share them. +- The default `./run` makes no outbound network requests. The opt-in `--llama` phase is the sole exception (it clones llama.cpp and may download a model); see below. Reports stay on the local disk unless the user chooses to share them. -## Optional GPU phase (`--gpu`) +## Code that runs beyond pure bash + Python -The default `./run` is pure bash plus Python stdlib and ships no compiled binaries. The opt-in `--gpu` phase is the one exception: a real GPU load needs a GPU kernel, so `gpu-variance.sh` compiles a small Metal compute kernel with `swiftc` at runtime. The full Metal and Swift source is inline in the script's heredoc, auditable the same way as everything else: read it before running. Nothing pre-compiled is shipped, the compiled binary lives only in a per-user tempfile deleted on exit, the kernel runs read-mostly FMA math over a scratch buffer, and the phase makes no network calls. If `swiftc` or a Metal device isn't available it skips cleanly. To keep the run fully interpreted, don't pass `--gpu`. +Three phases go past "interpreted, offline" scripts. The first is on by default but never touches the network; the other two are opt-in and off unless you ask for them. Nothing pre-compiled is ever shipped in the repo. + +- **Phase 12 memory bandwidth (default).** `memory-bandwidth.sh` prefers a vendored STREAM triad: it compiles `stream-triad.c` with the `clang` from the Command Line Tools, runs it, and deletes the binary. The C source is in the repo and short enough to read. No network, no shipped binary, no privileges. If `clang` is unavailable it falls back to the pure-Python `memmove` proxy. +- **`--gpu` (opt-in, offline).** `gpu-variance.sh` compiles a small Metal compute kernel with `swiftc` at runtime. The Metal and Swift source is inline in the script's heredoc; read it before running. The compiled binary lives in a per-user tempfile deleted on exit, the kernel runs read-mostly FMA math over a scratch buffer, and it makes no network calls. Skips cleanly if `swiftc` or a Metal device is unavailable. +- **`--llama` (opt-in, reaches the network).** This is the only phase that leaves the machine and runs third-party code. `llama-bench.sh` clones llama.cpp at a pinned ref into `~/.cache/shakedown/llama`, builds it with `cmake`, may download a small GGUF model (a pinned `LLAMA_MODEL_URL`, or a local `LLAMA_MODEL` you provide), and runs `llama-bench`. It is off unless you pass `--llama`, and it skips cleanly when `git` / `cmake` / the network / a model is unavailable. The clone is pinned and the cache is reused, so you can audit exactly what was fetched. The auto-download is not integrity-checked unless you set `LLAMA_MODEL_SHA256` (a complete but corrupted or tampered download would otherwise be cached and run); for a guaranteed-consistent model, point `LLAMA_MODEL` at a local `.gguf` you trust. To keep the run fully offline and free of third-party code, simply don't pass `--llama`. ## Privacy in submitted reports diff --git a/Verification/Benchmark Reference.md b/Verification/Benchmark Reference.md new file mode 100644 index 0000000..b9b9ec7 --- /dev/null +++ b/Verification/Benchmark Reference.md @@ -0,0 +1,142 @@ +--- +created: 2026-06-02 +tags: [verification, benchmarks, reference] +--- + +# Benchmark Reference + +Shakedown's own thresholds are not yet calibrated against confirmed-good hardware, and its default workload (SHA-256) is hardware-accelerated, so the harness is a structured *consistency* check, not an absolute pass/fail. Published third-party benchmarks fill the other half: an absolute floor your unit should clear when cool, plus a crowd-sourced distribution of real units you can compare against. This is the "install this, run this, your unit should score about X" companion to the [Runbook](Runbook.md), tuned for catching a bad unit (the M5 Max especially) in a store you cannot easily return to. + +Use it alongside the harness: +- `./run --store` runs the thorough profile (both CPU workloads, GPU, longer warmup, more iterations) for the structured variance and thermal verdict. +- `compare-reports.sh REFERENCE.json UNIT.json` diffs your unit against a known-good sibling, which is the most trustworthy signal while the absolute thresholds are uncalibrated. + +## How to read these numbers + +- **The links are the source of truth, the printed numbers are a snapshot.** Scores drift with macOS releases, firmware/SMC updates, ambient temperature, power mode, and config, and crowd averages move as submissions land. Re-check the live link for your exact SKU. Treat a printed range as a healthy floor the machine should reach when cool, not a target it must hold under sustained load. +- **Single-core is the cleanest tell.** It barely varies with cooling, chassis, or bin, so a single-core result more than ~5-7% below the band is the most reliable red flag for a real defect, a rebinned/wrong chip, background load, or a bad power mode. Multi-core is weaker evidence on its own: it swings with cooling, power mode, and core-count bin, so confirm you are comparing the right bin first. +- **Compare same chip AND same core-count bin AND same chassis.** A 14" M5 Max legitimately throttles well below a 16" by design. A fanless Air tapers under sustained load by design. Neither is a defect. +- **Benchmark hygiene.** "CB R24" here means Cinebench 2024 (Maxon dropped the "R"). Do not mix it with the older Cinebench R23 (Intel-era user reports, different scale) or with Macworld's Cinebench 2026 (a different benchmark again). +- **Confidence tiers:** *well-sourced* = cross-confirmed against primary sources; *rough* = real but thin or internally disagreeing, verify live; *look up live* = no trustworthy published number, do not trust a printed figure. + +## Tools (no Terminal or Xcode needed) + +- **Geekbench 6** (CPU and Metal GPU). Point-and-click, installs in a couple of minutes. The single biggest sample of real Mac scores. +- **Cinebench 2024** (Maxon). The sustained-loop test: it has a minimum-test-duration / loop mode that heat-soaks the chassis, which is what actually surfaces the M5 defect. +- **[mianibench.com](https://mianibench.com/)** (crowd-sourced, Apple-only). Real-world Cinebench plus Blender, Final Cut, DaVinci Resolve, and Cyberpunk results, filterable by chip series (M1 to M5) and variant (Standard / Pro / Max / Ultra). This is the best *distribution* view: it shows the spread of real units of your exact SKU, so you can see whether yours is a normal sample or a low outlier. That outlier check is the comparison the harness can't do on its own. (Aggregated views may want a free account; it is a browser app.) +- **Apple Diagnostics** (hold D, or Option-D, at power-on). Apple's own logic-board / sensor / fan / power self-test, with a reference code you can show staff. +- A browser display tester ([screentest.run](https://screentest.run/), [oledtest.org](https://oledtest.org/)) for dead/stuck pixels and bleed. + +## Live lookup (check your exact SKU before relying on anything) + +- [Geekbench 6 Mac benchmarks](https://browser.geekbench.com/mac-benchmarks): per-model averages, the live yardstick. Open in a normal browser (the pages 403 automated fetches). +- [mianibench.com Cinebench](https://mianibench.com/benchmark/1): crowd-sourced distribution by chip + variant. +- [Notebookcheck M5 Max](https://www.notebookcheck.net/Apple-M5-Max-Processor-Benchmarks-and-Specs.1244918.0.html) and the [M5 Pro / M5 Max CPU analysis](https://www.notebookcheck.net/Apple-M5-Pro-M5-Max-CPU-Analysis-M5-Max-is-not-much-faster-than-the-M4-Max.1246054.0.html) (includes the 16" vs 14" split). +- [MacRumors M5 Max leak](https://www.macrumors.com/2026/03/05/m5-max-geekbench-benchmarks/): GB6 single ~4,268 / multi ~29,233. + +## Reference scores by generation + +Cinebench numbers are Cinebench 2024 / R24. Geekbench numbers are Geekbench 6. Ranges are real samples, not targets. + +### Apple M5 family (2025-2026) + +Compare same chassis only: the 14" M5 Max legitimately throttles well below the 16" by design. + +| SKU | CB R24 multi | GB6 multi | GB6 single | Confidence | +|---|---|---|---|---| +| M5 (base, 10-core) MBP 14" / Air / iPad Pro | ~957-1,172 (MBP 14" tops the band, Air lower) | ~16,500-18,050 | ~4,130-4,330 | well-sourced | +| M5 Pro (18-core) MBP 14"/16" | ~2,347 (one 16" sample, approximate) | ~28,436 (thin samples) | ~4,295 | rough, verify live | +| **M5 Max (18-core CPU, 40-core GPU) MBP 16"** | **~2,073-2,437** (16" ~2,437, 14" ~2,073) | **~29,000-29,415** | **~4,200-4,335** | well-sourced | + +### Apple M4 family (2024-2025) + +| SKU | CB R24 multi | GB6 multi | GB6 single | Confidence | +|---|---|---|---|---| +| M4 (10-core) MBA / mini / iMac / MBP 14 base | ~815-986 (fanless Air lower) | ~14,650-15,200 | ~3,650-3,840 | well-sourced | +| M4 Pro (12-core) mini / MBP 14 | ~1,400-1,460 | ~20,300-20,700 | ~3,830-3,930 | well-sourced | +| M4 Pro (14-core) mini / MBP 14/16 | ~1,660-1,750 | ~22,400-22,600 | ~3,830-3,930 | well-sourced | +| M4 Max (14-core, 32-GPU) MBP 14/16 | ~1,700-1,800 (look up live) | ~22,000-23,500 (look up live) | ~3,850-3,950 | look up live | +| M4 Max (16-core, 40-GPU) MBP 14/16 | ~2,000-2,100 | ~25,600-26,700 | ~3,880-4,060 | well-sourced | + +### Apple M3 family (2023-2024) + +Single-core is near-flat family-wide (~3,050-3,160 GB6), so it is a health check, not a SKU differentiator. + +| SKU | CB R24 multi | GB6 multi | GB6 single | Confidence | +|---|---|---|---|---| +| M3 8-core (cooled) MBP 14 / iMac / mini | ~590-712 (treat ~600 as typical) | ~11,700-12,100 | ~3,000-3,130 | well-sourced | +| M3 8-core (fanless) MacBook Air | look up live (~580-600, then throttles) | ~11,500-12,100 | ~3,000-3,160 | look up live | +| M3 Pro 11-core | ~900-960 | ~14,000-14,500 | ~3,080-3,130 | well-sourced | +| M3 Pro 12-core | ~1,000-1,080 | ~15,000-15,500 | ~3,090-3,140 | well-sourced | +| M3 Max 14-core (30-GPU) | ~1,350-1,510 | ~18,500-19,000 | ~3,050-3,150 | well-sourced | +| M3 Max 16-core (40-GPU) | ~1,530-1,660 | ~20,900-21,300 | ~3,090-3,160 | well-sourced | + +### Apple M2 family (2022-2023) + +| SKU | CB R24 multi | GB6 multi | GB6 single | Confidence | +|---|---|---|---|---| +| M2 (8-core) MBA / MBP 13 / mini | ~555-600 | ~9,600-9,750 | ~2,580-2,650 | well-sourced | +| M2 Pro 10-core base MBP 14 / mini | look up live (~640-680) | ~12,300-12,900 | ~2,640-2,670 | rough | +| M2 Pro 12-core MBP 14/16 / mini | ~780-800 | ~14,400-14,600 | ~2,650-2,665 | well-sourced | +| M2 Max (12-core) MBP 14/16 / Studio | ~1,030-1,070 | ~14,600-14,900 | ~2,700-2,750 | well-sourced | + +### Apple M1 family (2020-2021) + +| SKU | CB R24 multi | GB6 multi | GB6 single | Confidence | +|---|---|---|---|---| +| M1 (8-core) MBA / MBP 13 / mini / iMac 24 | ~444-525 | ~8,200-8,650 | ~2,320-2,370 | well-sourced | +| M1 Pro 8-core MBP 14 base | ~628 | ~10,314 | ~2,360 | well-sourced | +| M1 Pro 10-core MBP 14/16 | ~824 | ~12,250-12,360 | ~2,370-2,390 | well-sourced | +| M1 Max 10-core MBP 14/16 / Studio | ~796 | ~12,277-12,661 | ~2,376-2,419 | well-sourced | + +### Intel MacBook Pro 16" (Late 2019) + +9th-gen Coffee Lake-H. Geekbench 6 (14k-16k Mac-specific samples) is the trustworthy yardstick. Cinebench R24 data for these chips is sparse, largely non-Mac, and internally inconsistent, so it is look-up-live only. + +| SKU | CB R24 multi | GB6 multi | GB6 single | Confidence | +|---|---|---|---|---| +| i7-9750H (6-core) base | look up live | ~5,000-5,500 | ~1,280-1,320 | well-sourced (GB6) | +| i9-9880H (8-core) common | look up live | ~6,000-6,500 | ~1,330-1,380 | well-sourced (GB6) | +| i9-9980HK (8-core) top BTO | look up live | ~6,200-6,700 | ~1,350-1,400 | rough | + +## Spotting a bad unit + +Three independent signals separate a good unit from a bad one. Weigh all three; a single low number is not proof. + +1. **Score vs baseline (single-core is the cleanest tell).** Run Geekbench 6 and Cinebench 2024 once on a cool, plugged-in machine and compare to the matching SKU row. A single-core result more than ~5-7% below the band is the most reliable red flag. A multi-core result 15-20%+ below baseline with no explanation (an M5 Max 16" under ~23-24k GB6 multi, an M4 Max under ~20k, an Intel i9-9880H near 5,000) is suspect once you have confirmed the bin. +2. **Run-to-run variance.** Run the benchmark 3-5 times back to back from a cold start. For the M5 Max specifically, run GB6 multi 5 times with ~30 s gaps and compute `(max - min) / mean`; flag if the spread exceeds ~10%. Healthy units cluster within a few percent and single-core varies only ~1-2%. Wild swings on a cool machine are the headline early-M5-Max bad-batch signature (reported up to ~41.5%), and they show in multi-core, not single. +3. **Drop under sustained heat (the most important test).** Loop Cinebench 2024 multi-core for ~10+ minutes and watch the score per pass. Normal: the sustained score settles ~10-15% below the cold-burst peak after the first 1-2 runs, then HOLDS steady with loud-but-steady fans. Not acceptable: a continuous downward staircase, a plateau far below the SKU band, fans pinned at max while the score still collapses, or a thermal shutdown / kernel panic. That monotonic decay is the signature of bad paste, poor heatsink contact, a weak fan, or a thermal-sensor fault. + +**SKU sanity check:** confirm System Information reports the advertised CPU and core/thread count. An M3 Max scoring like an M3 Pro, or a "9980HK" reporting 6 cores, is a relabeled or wrong unit. Walk away. + +**Do not false-flag:** fanless Airs (M1-M4) and the 14" M5 Max throttle under sustained load by design. Always compare same chip, same bin, same chassis. If a fan-equipped Pro throttles like a fanless Air, the cooling is suspect. + +## In the store (~15-20 min, catch the obvious before you leave) + +Cheapest and most decisive reads first, then one quick baseline, then a fast cosmetic sweep. + +1. **Power and Wi-Fi.** Plug into the AC charger (sustained benchmarks need wall power) and join store Wi-Fi. +2. **Instant battery read.** Option-click the Apple menu, then System Information, then Power. Cycle Count should be 0-2 and Condition Normal. High cycles or "Service Recommended" on a "new" unit is a reject. +3. **Apple Diagnostics.** Shut down, hold D at power-on. 2-5 min, surfaces logic-board / sensor / fan / power faults with a code you can show staff. +4. **Install Geekbench 6 + Cinebench 2024** while diagnostics run (GUI apps, no Terminal). +5. **One baseline run each.** Write down the numbers and compare to the SKU row above and to mianibench / Geekbench Browser. One pass does not clear a unit, but it catches an instant failure and gives a reference. +6. **Cosmetic + I/O sweep.** Lid/chassis creak, every port, both speakers, a solid-color display sweep for dead/stuck pixels, and listen for coil whine during the benchmark. +7. **The sibling control (highest leverage).** Benchmark a floor/demo unit of the SAME model and config under the same ambient. The defect is unit-level, so an A/B against a known-good twin is the cleanest judge: yours should land within a few percent on both single-run and sustained scores. Put the two panels side by side on the same test image. +8. **If anything is clearly off, exchange it right there** before leaving the counter. + +## That night at the hotel (the real test, still in the city) + +The 15-20 min store window cannot run a sustained loop. Do it that night, on AC, on a hard flat surface in a cool room. + +1. **Harness, thorough profile:** `SHAKEDOWN_YES=1 ./run --target mbp-16-m5-max-64 --store`. Both CPU workloads (including the non-accelerated BLAKE2b that matches where the defect was reported), the GPU pass, a long warmup, and 8 iterations. Rerun any single WARN. +2. **Sustained Cinebench loop:** Cinebench 2024 multi-core back to back for ~30-45 min (or its minimum-test-duration mode). Record EACH run's score. A healthy unit settles to a stable sustained score; a defective one keeps sagging run-over-run. Infant-mortality and batch-variance defects often pass a cold first run and only show by the 3rd-4th run under heat. +3. **Second workload:** re-run Geekbench 6 (CPU and Metal) a few times against the live Geekbench Browser average for your exact model. +4. **Compare against a sibling** if you captured one: `./Verification/scripts/compare-reports.sh sibling.json yours.json`. +5. **Unhurried display + I/O:** dark-room black/white/grey/RGB fields for pixels and bleed, every key, Touch ID, every port in both orientations, MagSafe, headphone jack, trackpad corners, speakers at volume, mics, webcam, Wi-Fi/Bluetooth range. +6. **If the night test reveals a defect**, you still have the next in-city day to exchange it. Keep the receipt and packaging until the unit fully passes. + +## Hong Kong return constraint (this drives the whole protocol) + +Apple Hong Kong retail purchases are effectively exchange-only and only for an Apple-verified defect (no change-of-mind returns), with at most a 14-day defective-exchange window and original receipt + packaging required. A bad unit has to be caught and exchanged **while you are still in Hong Kong**, ideally before leaving the store. After you fly out there is no easy return, only AppleCare warranty repair. Leave at least one buffer day in the city, and keep the receipt and all packaging until the unit has passed the night test. + +> Every printed score above is advisory and drifts over time. Confirm the SKU and core count, compare same chip / same bin / same chassis, prefer an A/B against a known-good sibling, and weigh all three signals (vs-baseline, run-to-run variance, sustained-load trend) together before rejecting a unit. The live-lookup links, not these numbers, are the truth. diff --git a/Verification/Runbook.md b/Verification/Runbook.md index 7ae2068..aeb6e5d 100644 --- a/Verification/Runbook.md +++ b/Verification/Runbook.md @@ -5,7 +5,7 @@ tags: [verification, runbook] # QA Runbook -The procedure for each verification phase. The automated phases (0–5, the cold benchmarks 10–12, and the opt-in 4b / 13) are run by [`./run`](../README.md#quick-start); phases 6–9 are manual checks you step through yourself. Each phase has a goal, an action, and a pass condition. See [Pass-Fail Criteria](Pass-Fail%20Criteria.md) for the consolidated thresholds and [the JSON report schema](../Reports/SCHEMA.md) for the canonical output format. +The procedure for each verification phase. The automated phases (0–5, the cold benchmarks 10–12, and the opt-in 4b / 13 / 14) are run by [`./run`](../README.md#quick-start); phases 6–9 are manual checks you step through yourself. Each phase has a goal, an action, and a pass condition. See [Pass-Fail Criteria](Pass-Fail%20Criteria.md) for the consolidated thresholds, [Benchmark Reference](Benchmark%20Reference.md) for published score baselines and the in-store / hotel protocol, and [the JSON report schema](../Reports/SCHEMA.md) for the canonical output format. **Total time:** ~45 min on a MacBook Pro, ~25 min on a fanless Air, ~35 min on a desktop or Intel Mac, +30 min if you opt into the idle-drain test. diff --git a/Verification/scripts/compare-reports.sh b/Verification/scripts/compare-reports.sh new file mode 100755 index 0000000..c44c3f6 --- /dev/null +++ b/Verification/scripts/compare-reports.sh @@ -0,0 +1,152 @@ +#!/bin/bash +# compare-reports.sh: side-by-side comparison of two Shakedown reports. +# +# The harness's absolute thresholds are not yet calibrated against confirmed-good +# hardware, but the M5-class defect is unit-level: a good sibling of the same SKU +# runs fine. So the most trustworthy in-store signal is comparison, not an +# absolute pass/fail. Run the same ./run on a known-good unit (or get a friend's +# report for your exact SKU) and diff it against the unit you are verifying. +# +# Usage: +# ./Verification/scripts/compare-reports.sh REFERENCE.json UNIT.json +# +# REFERENCE.json a known-good report for the same SKU (the baseline) +# UNIT.json the report for the unit you are verifying +# +# Output: a human-readable table to stdout. It flags every metric where the unit +# materially trails the reference (slower throughput, wider variance, hotter, +# steeper cliff). Flags are advisory: a single flag warrants a rerun, several +# flags on the same unit mean it is the weaker silicon. Comparison is only +# meaningful within the same chip + memory SKU; a mismatch is called out loudly. + +set -euo pipefail + +if [[ $# -ne 2 ]]; then + echo "usage: $(basename "$0") REFERENCE.json UNIT.json" >&2 + echo " REFERENCE = a known-good report for the same SKU; UNIT = the one you are verifying" >&2 + exit 2 +fi + +for f in "$1" "$2"; do + if [[ ! -f "$f" ]]; then + echo "compare-reports.sh: file not found: $f" >&2 + exit 2 + fi +done + +python3 - "$1" "$2" <<'PYEOF' +import json +import sys + + +def load(path): + with open(path) as f: + return json.load(f) + + +ref = load(sys.argv[1]) +unit = load(sys.argv[2]) +ref_name = sys.argv[1].rsplit("/", 1)[-1] +unit_name = sys.argv[2].rsplit("/", 1)[-1] + + +def get(report, phase_key, path): + node = report.get("phases", {}).get(phase_key, {}).get("details", {}) + for key in path: + if not isinstance(node, dict): + return None + node = node.get(key) + return node if isinstance(node, (int, float)) else None + + +def uinfo(report): + u = report.get("unit", {}) or {} + return u.get("chip"), u.get("memory_gb") + + +# (label, phase_key, details-path, higher_is_better, kind, flag_threshold) +# kind: rel = percent change; pp = percentage-point delta; ratio = absolute +# delta on a ratio; abs = absolute delta (degrees C). +SPECS = [ + ("CPU variance mean (MB/s)", "4_cpu_variance", ["mean_mb_per_s"], True, "rel", 10), + ("CPU variance spread (%)", "4_cpu_variance", ["spread_pct"], False, "pp", 3), + ("CPU max/min ratio", "4_cpu_variance", ["max_to_min_ratio"], False, "ratio", 0.1), + ("Non-accel mean (MB/s)", "4b_cpu_variance_noaccel", ["mean_mb_per_s"], True, "rel", 10), + ("Non-accel spread (%)", "4b_cpu_variance_noaccel", ["spread_pct"], False, "pp", 3), + ("Race xz (MB/s)", "10_race_bench", ["throughput_mb_per_s"], True, "rel", 10), + ("SSD read (MB/s)", "11_ssd_test", ["read_mb_per_s"], True, "rel", 12), + ("SSD write (MB/s)", "11_ssd_test", ["write_mb_per_s"], True, "rel", 12), + ("Memory triad (GB/s)", "12_memory_bandwidth", ["mean_triad_gb_per_s"], True, "rel", 10), + ("Memory copy (GB/s)", "12_memory_bandwidth", ["mean_copy_gb_per_s"], True, "rel", 10), + ("GPU compute (GFLOP/s)", "13_gpu_variance", ["mean_gflops"], True, "rel", 10), + ("llama gen (tok/s)", "14_llama_bench", ["gen_tok_per_s"], True, "rel", 10), + ("llama prompt (tok/s)", "14_llama_bench", ["prompt_tok_per_s"], True, "rel", 10), + ("Thermal steady vs peak (%)", "5_thermal_load", ["steady_state_vs_peak_pct"], True, "pp", 5), + ("Thermal early cliff (%)", "5_thermal_load", ["early_cliff_pct"], False, "pp", 10), + ("CPU temp max (C)", "5_thermal_load", ["cpu_die_temp_c", "max"], False, "abs", 5), +] + + +def fmt(v): + if v is None: + return "n/a" + return f"{v:,.2f}" if isinstance(v, float) else f"{v:,}" + + +rows = [] +flags = [] +for label, pk, path, higher, kind, thr in SPECS: + a = get(ref, pk, path) + b = get(unit, pk, path) + delta_str = "n/a" + flagged = False + if a is not None and b is not None: + if kind == "rel": + d = (b - a) / a * 100 if a else 0.0 + delta_str = f"{d:+.1f}%" + flagged = (higher and d <= -thr) or (not higher and d >= thr) + elif kind == "pp": + d = b - a + delta_str = f"{d:+.1f}pp" + flagged = (higher and d <= -thr) or (not higher and d >= thr) + elif kind == "ratio": + d = b - a + delta_str = f"{d:+.2f}" + flagged = (b >= a + thr) and (b >= 1.2) + elif kind == "abs": + d = b - a + delta_str = f"{d:+.1f}" + flagged = (not higher and d >= thr) or (higher and d <= -thr) + rows.append((label, fmt(a), fmt(b), delta_str, "<-- unit trails" if flagged else "")) + if flagged: + flags.append(label) + +ref_chip, ref_mem = uinfo(ref) +unit_chip, unit_mem = uinfo(unit) + +print(f"shakedown compare") +print(f" reference: {ref_name} [{ref_chip or '?'}, {ref_mem or '?'} GB]") +print(f" unit: {unit_name} [{unit_chip or '?'}, {unit_mem or '?'} GB]") +if (ref_chip and unit_chip and ref_chip != unit_chip) or (ref_mem and unit_mem and ref_mem != unit_mem): + print() + print(" WARNING: different SKU (chip or memory). Comparison is only meaningful") + print(" within the same chip + memory config. Deltas below are not trustworthy.") +print() +print(f" {'metric':<28} {'reference':>12} {'unit':>12} {'delta':>9} note") +print(f" {'-'*28} {'-'*12} {'-'*12} {'-'*9} {'-'*16}") +for label, a, b, d, note in rows: + print(f" {label:<28} {a:>12} {b:>12} {d:>9} {note}") +print() +if flags: + print(f" Unit trails the reference on {len(flags)} metric(s): {', '.join(flags)}.") + if len(flags) == 1: + print(" One flag: rerun the unit (single-warn discipline) before reading into it.") + else: + print(" Several flags on the same unit point at the weaker silicon. Investigate,") + print(" rerun, and prefer the stronger unit while you can still exchange.") +else: + print(" No metric materially trails the reference: the unit looks consistent with") + print(" a known-good sibling of this SKU (within the comparison thresholds).") +print() +print(" Advisory only: thresholds compare relative health, not absolute pass/fail.") +PYEOF diff --git a/Verification/scripts/llama-bench.sh b/Verification/scripts/llama-bench.sh new file mode 100755 index 0000000..d710f6e --- /dev/null +++ b/Verification/scripts/llama-bench.sh @@ -0,0 +1,231 @@ +#!/bin/bash +# llama-bench.sh: OPT-IN combined CPU + GPU + memory load via llama.cpp. +# +# This is the only phase that reaches the network and runs third-party code. It +# is off unless you pass `./run --llama`, and it degrades to a "skipped" verdict +# (never failing the run) at every step. It exists because the M5 Max defect was +# reported on AI workloads, and an LLM inference benchmark is the one test that +# stresses the CPU, the GPU (Metal), and the memory subsystem together under +# sustained load, which none of the built-in phases do. +# +# What it does, opt-in only: +# 1. git clone llama.cpp (ggml-org) at a pinned ref into a cache dir +# 2. build llama-bench with cmake (Metal + Accelerate, both ship with macOS) +# 3. obtain a small GGUF model (LLAMA_MODEL path, or download LLAMA_MODEL_URL) +# 4. run `llama-bench` and record prompt/generation tokens-per-second + spread +# Each step skips cleanly if its tool / network / model is unavailable. The clone +# and build are cached so reruns are fast. See SECURITY.md for the disclosure. +# +# Output: JSON to stdout. Progress to stderr. +# +# Env knobs: +# LLAMA_REPO default https://github.com/ggml-org/llama.cpp +# LLAMA_REF default a pinned tag (override to bump); falls back to the +# default branch with a recorded note if the tag is missing +# LLAMA_REPS default 5 (llama-bench repetitions, for run-to-run spread) +# LLAMA_MODEL path to a local .gguf (preferred; skips the download) +# LLAMA_MODEL_URL default small GGUF to download if LLAMA_MODEL is unset +# LLAMA_MODEL_SHA256 optional. If set, the downloaded model is verified against +# this sha256 before it is cached; a mismatch skips the phase. +# Unset means the download is NOT integrity-checked (see +# SECURITY.md); provide a local LLAMA_MODEL for guaranteed +# consistency. +# SHAKEDOWN_LLAMA_DIR cache dir (default ~/.cache/shakedown/llama) + +set -euo pipefail + +LLAMA_REPO=${LLAMA_REPO:-https://github.com/ggml-org/llama.cpp} +LLAMA_REF=${LLAMA_REF:-b4585} +LLAMA_REPS=${LLAMA_REPS:-5} +LLAMA_DIR=${SHAKEDOWN_LLAMA_DIR:-$HOME/.cache/shakedown/llama} +LLAMA_MODEL=${LLAMA_MODEL:-} +LLAMA_MODEL_URL=${LLAMA_MODEL_URL:-https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf} +LLAMA_MODEL_SHA256=${LLAMA_MODEL_SHA256:-} + +if ! [[ "$LLAMA_REPS" =~ ^[0-9]+$ ]] || (( LLAMA_REPS < 1 )); then + echo "llama-bench.sh: LLAMA_REPS must be a positive integer (got '$LLAMA_REPS')" >&2 + exit 2 +fi + +emit_skipped() { + # $1: human-readable reason. Routed through python3 so the JSON is well-formed. + python3 - "$1" <<'PYEOF' +import json +import sys +reason = sys.argv[1] +print(json.dumps({ + "workload": "llama.cpp llama-bench (combined CPU + GPU + memory, AI inference)", + "verdict": "skipped", + "data_quality": "skipped", + "data_quality_notes": [reason], + "verdict_reasons": [reason + "; phase skipped"], +}, indent=2)) +PYEOF +} + +if ! command -v git >/dev/null 2>&1; then + emit_skipped "git not found (the --llama phase needs git to clone llama.cpp)" + exit 0 +fi +if ! command -v cmake >/dev/null 2>&1; then + emit_skipped "cmake not found (the --llama phase builds llama.cpp with cmake; install it, e.g. brew install cmake, to enable this opt-in phase)" + exit 0 +fi + +mkdir -p "$LLAMA_DIR" +SRC="$LLAMA_DIR/llama.cpp" +BIN="$SRC/build/bin/llama-bench" +ref_note="" + +# 1) Clone (cached). Prefer the pinned ref; fall back to the default branch. +if [[ ! -d "$SRC/.git" ]]; then + echo "llama-bench: cloning $LLAMA_REPO @ $LLAMA_REF (one-time, cached in $LLAMA_DIR)" >&2 + if ! git clone --depth 1 --branch "$LLAMA_REF" "$LLAMA_REPO" "$SRC" 2>/dev/null; then + if ! git clone --depth 1 "$LLAMA_REPO" "$SRC" 2>/dev/null; then + emit_skipped "git clone failed (offline, or $LLAMA_REPO unreachable); set LLAMA_MODEL and retry on a network" + exit 0 + fi + ref_note="pinned ref $LLAMA_REF not found; built from the default branch instead" + fi +fi + +# 2) Build llama-bench (cached). Metal + Accelerate are used automatically on macOS. +if [[ ! -x "$BIN" ]]; then + echo "llama-bench: building llama-bench with cmake (one-time, ~3-5 min)" >&2 + if ! cmake -S "$SRC" -B "$SRC/build" -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1; then + emit_skipped "cmake configure failed" + exit 0 + fi + if ! cmake --build "$SRC/build" --target llama-bench -j >/dev/null 2>&1; then + emit_skipped "llama.cpp build failed" + exit 0 + fi +fi +if [[ ! -x "$BIN" ]]; then + emit_skipped "build produced no llama-bench binary at $BIN" + exit 0 +fi + +# 3) Model: prefer a user-provided path, else download the pinned small GGUF. +MODEL="$LLAMA_MODEL" +if [[ -z "$MODEL" ]]; then + MODEL="$LLAMA_DIR/model.gguf" + if [[ ! -f "$MODEL" ]]; then + echo "llama-bench: downloading model (one-time, cached): $LLAMA_MODEL_URL" >&2 + if ! curl -fL --retry 2 -o "$MODEL.partial" "$LLAMA_MODEL_URL" 2>/dev/null; then + rm -f "$MODEL.partial" + emit_skipped "model download failed; set LLAMA_MODEL=/path/to/model.gguf (any small GGUF) and rerun" + exit 0 + fi + # Optional integrity check. A complete but corrupted or tampered download + # would otherwise be cached and run, so verify when a hash is provided. + if [[ -n "$LLAMA_MODEL_SHA256" ]]; then + got=$(shasum -a 256 "$MODEL.partial" 2>/dev/null | awk '{print $1}') + if [[ "$got" != "$LLAMA_MODEL_SHA256" ]]; then + rm -f "$MODEL.partial" + emit_skipped "model checksum mismatch (got ${got:-none}, expected $LLAMA_MODEL_SHA256); refusing to cache" + exit 0 + fi + fi + mv "$MODEL.partial" "$MODEL" + fi +fi +if [[ ! -f "$MODEL" ]]; then + emit_skipped "no model file available; set LLAMA_MODEL=/path/to/model.gguf" + exit 0 +fi + +# 4) Run the benchmark. -o json gives a stable, parseable result. +echo "llama-bench: running ($LLAMA_REPS reps, model $(basename "$MODEL"))" >&2 +T0=$(date +%s) +if ! RAW=$("$BIN" -m "$MODEL" -p 512 -n 128 -r "$LLAMA_REPS" -o json 2>/dev/null); then + emit_skipped "llama-bench run failed (model incompatible or out of memory?)" + exit 0 +fi +T1=$(date +%s) + +python3 - "$RAW" "$((T1 - T0))" "$MODEL" "$LLAMA_REF" "$ref_note" <<'PYEOF' +import json +import sys + +try: + rows = json.loads(sys.argv[1]) +except json.JSONDecodeError: + print(json.dumps({ + "workload": "llama.cpp llama-bench (combined CPU + GPU + memory, AI inference)", + "verdict": "skipped", + "data_quality": "skipped", + "data_quality_notes": ["llama-bench produced unparseable output"], + "verdict_reasons": ["llama-bench output not JSON; phase skipped"], + }, indent=2)) + sys.exit(0) + +wall = int(sys.argv[2]) +model = sys.argv[3].rsplit("/", 1)[-1] +ref = sys.argv[4] +ref_note = sys.argv[5] + +backend = None +prompt = None # (avg_ts, stddev_ts) +gen = None +for e in rows if isinstance(rows, list) else []: + backend = e.get("backend_name") or e.get("backend") or backend + n_gen = e.get("n_gen", 0) or 0 + n_prompt = e.get("n_prompt", 0) or 0 + avg = e.get("avg_ts") + std = e.get("stddev_ts", 0) or 0 + if avg is None: + continue + if n_prompt and not n_gen: + prompt = (avg, std) + elif n_gen and not n_prompt: + gen = (avg, std) + +if prompt is None and gen is None: + print(json.dumps({ + "workload": "llama.cpp llama-bench (combined CPU + GPU + memory, AI inference)", + "verdict": "skipped", + "data_quality": "skipped", + "data_quality_notes": ["llama-bench returned no prompt/generation rows"], + "verdict_reasons": ["no usable llama-bench result; phase skipped"], + }, indent=2)) + sys.exit(0) + + +def spread(t): + return round(t[1] / t[0] * 100, 2) if t and t[0] else None + + +notes = [] +if ref_note: + notes.append(ref_note) + +summary = [] +if prompt: + summary.append(f"prompt {prompt[0]:,.1f} tok/s") +if gen: + summary.append(f"gen {gen[0]:,.1f} tok/s") + +result = { + "workload": "llama.cpp llama-bench (combined CPU + GPU + memory, AI inference)", + "model": model, + "llama_ref": ref, + "backend": backend, + "prompt_tok_per_s": round(prompt[0], 2) if prompt else None, + "prompt_tok_per_s_stddev": round(prompt[1], 2) if prompt else None, + "prompt_spread_pct": spread(prompt), + "gen_tok_per_s": round(gen[0], 2) if gen else None, + "gen_tok_per_s_stddev": round(gen[1], 2) if gen else None, + "gen_spread_pct": spread(gen), + "wall_seconds": wall, + "data_quality": "ok", + "data_quality_notes": notes, + "verdict": "info", + "verdict_reasons": [ + f"{backend or 'llama.cpp'}: " + ", ".join(summary) + + (f" (gen spread {spread(gen):.1f}%)" if gen and spread(gen) is not None else ""), + "informational in v0.2; GPU/AI thresholds land once the corpus has baselines" + ], +} +print(json.dumps(result, indent=2)) +PYEOF diff --git a/Verification/scripts/memory-bandwidth.sh b/Verification/scripts/memory-bandwidth.sh index dc68575..577a6f0 100755 --- a/Verification/scripts/memory-bandwidth.sh +++ b/Verification/scripts/memory-bandwidth.sh @@ -1,30 +1,25 @@ #!/bin/bash -# memory-bandwidth.sh: STREAM-style sequential memory copy bandwidth. +# memory-bandwidth.sh: STREAM-style memory bandwidth. # -# Spawns one worker per P-core, each copying its own pair of large buffers with -# ctypes.memmove in a tight loop, and sums the aggregate throughput. A single -# thread cannot saturate the multi-channel memory controllers on Apple Silicon -# (a lone memcpy tops out well below peak), so the test is multi-threaded the way -# STREAM is built with OpenMP. +# Prefers the vendored STREAM triad (stream-triad.c: real Copy / Scale / Add / +# Triad across one thread per P-core), compiled at runtime with the clang that +# ships in the Xcode Command Line Tools. No network, no shipped binary: the C +# source is in the repo, compiled to a tempfile, run, deleted. When clang is +# unavailable it falls back to a pure-Python ctypes.memmove copy proxy, so the +# phase always produces a number. # -# What it measures: sequential copy bandwidth (memcpy). Each copy reads N bytes -# and writes N bytes, so "touched" traffic is 2x the reported copy bandwidth. -# This is a lower bound on a full STREAM triad (copy / scale / add / triad); -# scale and add need elementwise arithmetic that pure-Python can't do at memory -# speed without a native kernel. memmove is the honest pure-stdlib proxy: it -# still catches a memory subsystem that is slow or inconsistent across runs. -# -# Buffers are sized to exceed the last-level / system-level cache so the copy is -# DRAM-bound, not cache-bound. +# What it measures: DRAM bandwidth. A single thread cannot saturate the +# multi-channel controllers on Apple Silicon, so the kernels are multi-threaded +# the way STREAM uses OpenMP. Buffers are sized past the system-level cache. # # Output: JSON to stdout. Progress to stderr. # # Env knobs: -# MEMBW_MB default 1024 (TARGET total working set in MB across all -# workers; split into 2 buffers per worker. -# Auto-capped to fit RAM.) -# MEMBW_ITERS default 5 (timed iterations, for run-to-run variance) -# MEMBW_SECONDS default 2 (seconds per timed iteration) +# MEMBW_MB default 1024 (TARGET total working set in MB, auto-capped to +# 40% of RAM; split across 3 arrays for the triad +# or 2 buffers per worker for the proxy) +# MEMBW_ITERS default 5 (reps, for run-to-run variance) +# MEMBW_SECONDS default 2 (seconds per iteration, memmove proxy only) # WORKERS default = P-cores set -euo pipefail @@ -53,10 +48,93 @@ if ! [[ "$WORKERS" =~ ^[0-9]+$ ]] || (( WORKERS < 1 )); then fi MEMSIZE=$(sysctl -n hw.memsize 2>/dev/null || echo 0) +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +STREAM_SRC="$SCRIPT_DIR/stream-triad.c" + +# Budget the working set at 40% of physical RAM so we never push into swap. +budget_mb=$MEMBW_MB +if [[ "$MEMSIZE" =~ ^[0-9]+$ ]] && (( MEMSIZE > 0 )); then + ram_mb=$(( MEMSIZE / 1024 / 1024 )) + cap_mb=$(( ram_mb * 40 / 100 )) + if (( budget_mb > cap_mb )); then budget_mb=$cap_mb; fi +fi + +# ---- Preferred path: vendored STREAM triad compiled with clang ---- +if command -v clang >/dev/null 2>&1 && [[ -f "$STREAM_SRC" ]]; then + per_array_mb=$(( budget_mb / 3 )) + if (( per_array_mb >= 64 )); then + STREAM_BIN=$(mktemp -t shakedown-stream.XXXXXX) + trap 'rm -f "$STREAM_BIN"' EXIT INT TERM + if clang -O3 -pthread "$STREAM_SRC" -o "$STREAM_BIN" 2>/dev/null; then + per_array_bytes=$(( per_array_mb * 1024 * 1024 )) + echo "memory-bandwidth: STREAM triad, ${per_array_mb}MB/array x3, $WORKERS threads, $MEMBW_ITERS reps" >&2 + T0=$(date +%s) + if RAW=$("$STREAM_BIN" "$per_array_bytes" "$WORKERS" "$MEMBW_ITERS" 2>/dev/null); then + T1=$(date +%s) + python3 - "$RAW" "$WORKERS" "$per_array_mb" "$((T1 - T0))" <<'PYEOF' +import json +import statistics +import sys + +raw = json.loads(sys.argv[1]) +workers = int(sys.argv[2]) +per_array_mb = int(sys.argv[3]) +wall = int(sys.argv[4]) +res = raw["results"] + + +def mean(xs): + return round(statistics.mean(xs), 2) if xs else 0.0 + + +triad = res["triad"] +t_mean = mean(triad) +t_min = round(min(triad), 2) if triad else 0.0 +t_max = round(max(triad), 2) if triad else 0.0 +spread = round((t_max - t_min) / t_mean * 100, 2) if t_mean else 0.0 +c_mean, s_mean, a_mean = mean(res["copy"]), mean(res["scale"]), mean(res["add"]) + +result = { + "workload": "STREAM-style triad (vendored C, pthreads), DRAM-bound", + "method": "stream-triad", + "workers": workers, + "working_set_mb": per_array_mb * 3, + "reps": raw.get("reps"), + "copy_gb_per_s": res["copy"], + "scale_gb_per_s": res["scale"], + "add_gb_per_s": res["add"], + "triad_gb_per_s": triad, + "mean_copy_gb_per_s": c_mean, + "mean_scale_gb_per_s": s_mean, + "mean_add_gb_per_s": a_mean, + "mean_triad_gb_per_s": t_mean, + "min_triad_gb_per_s": t_min, + "max_triad_gb_per_s": t_max, + "spread_pct": spread, + "wall_seconds": wall, + "data_quality": "ok", + "data_quality_notes": [], + "verdict": "info", + "verdict_reasons": [ + f"triad mean {t_mean:,.1f} GB/s (copy {c_mean:,.1f}, scale {s_mean:,.1f}, " + f"add {a_mean:,.1f}) across {workers} threads, spread {spread:.1f}%", + "informational in v0.2; calibrated pass/fail thresholds land in v0.3" + ], +} +print(json.dumps(result, indent=2)) +PYEOF + exit 0 + fi + fi + fi + echo "memory-bandwidth: STREAM triad unavailable (clang/compile/size), using memmove proxy" >&2 + TRIAD_FALLBACK="STREAM triad unavailable (clang missing, compile failed, or too little RAM); used the pure-Python memmove proxy instead" +fi -echo "memory-bandwidth: ${MEMBW_MB}MB working set across $WORKERS workers, $MEMBW_ITERS x ${MEMBW_SECONDS}s" >&2 +# ---- Fallback: pure-Python ctypes.memmove copy proxy ---- +echo "memory-bandwidth: ${MEMBW_MB}MB working set across $WORKERS workers (memmove proxy)" >&2 -python3 - "$MEMBW_MB" "$MEMBW_ITERS" "$MEMBW_SECONDS" "$WORKERS" "$MEMSIZE" <<'PYEOF' +python3 - "$MEMBW_MB" "$MEMBW_ITERS" "$MEMBW_SECONDS" "$WORKERS" "$MEMSIZE" "${TRIAD_FALLBACK:-}" <<'PYEOF' import ctypes import json import multiprocessing @@ -73,19 +151,14 @@ ITERATIONS = int(sys.argv[2]) SECONDS = int(sys.argv[3]) WORKERS = int(sys.argv[4]) MEMSIZE_BYTES = int(sys.argv[5]) +FALLBACK_NOTE = sys.argv[6] MB = 1024 * 1024 -data_quality_notes = [] +data_quality_notes = [FALLBACK_NOTE] if FALLBACK_NOTE else [] -# Size each worker's buffers. Total working set is split into 2 buffers (src+dst) -# per worker. Floor per-buffer at 64 MB so it comfortably exceeds per-core L2 and -# the aggregate exceeds the system-level cache (DRAM-bound, not cache-bound). MIN_BUFFER_MB = 64 per_buffer_mb = max(MIN_BUFFER_MB, TARGET_MB // (2 * WORKERS)) -# RAM guard: keep the working set under 40% of physical memory so we don't push -# the machine into swap (which would measure the SSD, not RAM). Scale down, and -# skip only if we cannot keep buffers above a cache-busting 32 MB. if MEMSIZE_BYTES > 0: budget_mb = int(MEMSIZE_BYTES / MB * 0.40) max_buffer_mb = budget_mb // (2 * WORKERS) @@ -98,13 +171,13 @@ if MEMSIZE_BYTES > 0: if per_buffer_mb < 32: print(json.dumps({ "workload": "multi-threaded sequential memcpy (ctypes.memmove)", + "method": "memmove-proxy", "workers": WORKERS, "verdict": "skipped", "data_quality": "skipped", - "data_quality_notes": [ + "data_quality_notes": data_quality_notes + [ f"not enough RAM to size cache-busting buffers for {WORKERS} " - f"workers ({MEMSIZE_BYTES // MB} MB total); lower WORKERS or " - f"raise MEMBW_MB on a machine with more memory" + f"workers ({MEMSIZE_BYTES // MB} MB total)" ], "verdict_reasons": ["insufficient RAM for a DRAM-bound test; phase skipped"], }, indent=2)) @@ -120,7 +193,6 @@ def worker(args): dst = bytearray(per_bytes) csrc = (ctypes.c_char * per_bytes).from_buffer(src) cdst = (ctypes.c_char * per_bytes).from_buffer(dst) - # Fault every page in and prime, OUTSIDE the timed region. ctypes.memset(csrc, 1, per_bytes) ctypes.memset(cdst, 2, per_bytes) memmove = ctypes.memmove @@ -139,9 +211,8 @@ def run_window(seconds): with multiprocessing.Pool(WORKERS) as pool: results = pool.map(worker, [(per_buffer_bytes, seconds)] * WORKERS) total_bytes = sum(r[0] for r in results) - # Workers run concurrently; the longest worker bounds the wall window. wall = max((r[1] for r in results), default=0.0) - copy_bw = total_bytes / wall if wall > 0 else 0.0 # bytes copied/s (N per memmove) + copy_bw = total_bytes / wall if wall > 0 else 0.0 return copy_bw, wall @@ -170,6 +241,7 @@ if per_buffer_mb < MIN_BUFFER_MB: result = { "workload": "multi-threaded sequential memcpy (ctypes.memmove), DRAM-bound", + "method": "memmove-proxy", "workers": WORKERS, "per_buffer_mb": per_buffer_mb, "total_working_set_mb": working_set_mb, diff --git a/Verification/scripts/run-shakedown.sh b/Verification/scripts/run-shakedown.sh index 669a7b2..22db4fe 100755 --- a/Verification/scripts/run-shakedown.sh +++ b/Verification/scripts/run-shakedown.sh @@ -23,6 +23,8 @@ NOTES="" NO_SUDO=0 RUN_NOACCEL=0 RUN_GPU=0 +STORE=0 +RUN_LLAMA=0 while [[ $# -gt 0 ]]; do case "$1" in @@ -46,9 +48,17 @@ while [[ $# -gt 0 ]]; do RUN_GPU=1 shift ;; + --store) + STORE=1 + shift + ;; + --llama) + RUN_LLAMA=1 + shift + ;; -h|--help) cat <] [--no-sudo] [--noaccel] [--gpu] [--notes "free-form notes"] +Usage: $(basename "$0") [--target ] [--store] [--no-sudo] [--noaccel] [--gpu] [--llama] [--notes "free-form notes"] --target optional. Target preset name (file under targets/ without .json) e.g. mbp-16-m5-max-64, macbook-air-m5-16, mbp-16-intel-2019. @@ -63,6 +73,16 @@ Usage: $(basename "$0") [--target ] [--no-sudo] [--noaccel] [--gpu] [--n --gpu also run Phase 13: a Metal GPU compute variance pass. Compiles a small Metal kernel with swiftc at runtime; skips cleanly if swiftc or a Metal device is unavailable. + --llama also run Phase 14: an opt-in combined CPU+GPU+memory load. + Clones and builds llama.cpp (needs git + cmake + network) + and runs llama-bench. Reaches the network and runs + third-party code (see SECURITY.md). Skips cleanly if any of + git / cmake / network / model is unavailable. + --store thorough "paranoid" profile for verifying a new unit (M5 + especially). Turns on --noaccel and --gpu and runs a longer + warmup with more iterations, so an intermittent batch defect + has more chances to surface. ~40-50 min on AC. Honors any + WARMUP_SEC / ITERATIONS / SECONDS_PER_ITER you set yourself. --notes "..." optional free-form note to embed in the report. Setting any note flips submission_safe to false (notes may contain PII). @@ -78,6 +98,22 @@ HELP esac done +# --store: thorough profile for verifying a new unit (especially M5). Run both CPU +# workloads (SHA-256 plus the non-accelerated BLAKE2b that matches where the M5 +# defect was reported) and the GPU pass, with a longer warmup and more iterations +# so an intermittent batch defect has more chances to surface. The := assignments +# honor any timing env vars the user set explicitly. Exported so the cpu-variance +# child phases inherit them. +if [[ "$STORE" -eq 1 ]]; then + RUN_NOACCEL=1 + RUN_GPU=1 + : "${WARMUP_SEC:=420}" + : "${ITERATIONS:=8}" + : "${SECONDS_PER_ITER:=60}" + : "${NOACCEL_WARMUP_SEC:=180}" + export WARMUP_SEC ITERATIONS SECONDS_PER_ITER NOACCEL_WARMUP_SEC +fi + TARGET_FILE="" if [[ -n "$TARGET" ]]; then TARGET_FILE="$REPO_ROOT/targets/$TARGET.json" @@ -230,10 +266,15 @@ else fi if [[ -z "${SHAKEDOWN_YES:-}" ]]; then - duration_hint="race + SSD + memory benchmarks (~2 min), then Phase 4 variance (~8 min)" - if [[ "$NO_SUDO" -ne 1 ]]; then duration_hint="$duration_hint, Phase 5 thermal (~10 min)"; fi - if [[ "$RUN_NOACCEL" -eq 1 ]]; then duration_hint="$duration_hint, Phase 4b non-accelerated variance (~6 min)"; fi - if [[ "$RUN_GPU" -eq 1 ]]; then duration_hint="$duration_hint, Phase 13 GPU compute (~1 min)"; fi + if [[ "$STORE" -eq 1 ]]; then + duration_hint="STORE profile (thorough): both CPU workloads plus GPU, longer warmup and $ITERATIONS iterations, roughly 40-50 min total on AC" + else + duration_hint="race + SSD + memory benchmarks (~2 min), then Phase 4 variance (~8 min)" + if [[ "$NO_SUDO" -ne 1 ]]; then duration_hint="$duration_hint, Phase 5 thermal (~10 min)"; fi + if [[ "$RUN_NOACCEL" -eq 1 ]]; then duration_hint="$duration_hint, Phase 4b non-accelerated variance (~6 min)"; fi + if [[ "$RUN_GPU" -eq 1 ]]; then duration_hint="$duration_hint, Phase 13 GPU compute (~1 min)"; fi + fi + if [[ "$RUN_LLAMA" -eq 1 ]]; then duration_hint="$duration_hint; plus Phase 14 llama.cpp (opt-in: clones + builds, can add 5-15 min on first run)"; fi cat <&2 About to run $duration_hint. Fans will spin up loud and the chassis will get @@ -279,6 +320,7 @@ VARIANCE_JSON="$WORK/variance.json" NOACCEL_JSON="$WORK/noaccel.json" THERMAL_JSON="$WORK/thermal.json" GPU_JSON="$WORK/gpu.json" +LLAMA_JSON="$WORK/llama.json" ignite "Phase 0: preflight" { @@ -358,6 +400,18 @@ else EOF fi +if [[ "$RUN_LLAMA" -eq 1 ]]; then + ignite "Phase 14: llama.cpp combined load (opt-in, clones + builds)" + start_heartbeat + "$SCRIPT_DIR/llama-bench.sh" > "$LLAMA_JSON" + stop_heartbeat +else + ignite "Phase 14: skipped (opt-in; pass --llama)" + cat > "$LLAMA_JSON" < + * Output: one JSON object on stdout with per-rep bandwidth (GB/s) per kernel. + * + * A single thread cannot saturate the multi-channel memory controllers on + * Apple Silicon, so the kernels are split across worker threads the + * way STREAM uses OpenMP. Arrays should be sized well past the system-level + * cache so the result is DRAM-bound, not cache-bound. + */ +#include +#include +#include +#include +#include + +static double *a, *b, *c; +static size_t N; /* elements per array */ +static int NTHREADS; +static const double SCALAR = 3.0; + +typedef struct { size_t lo, hi; int kernel; } work_t; + +static void *run(void *arg) { + work_t *w = (work_t *)arg; + size_t lo = w->lo, hi = w->hi; + switch (w->kernel) { + case 0: for (size_t i = lo; i < hi; i++) c[i] = a[i]; break; /* Copy */ + case 1: for (size_t i = lo; i < hi; i++) b[i] = SCALAR * c[i]; break; /* Scale */ + case 2: for (size_t i = lo; i < hi; i++) c[i] = a[i] + b[i]; break; /* Add */ + case 3: for (size_t i = lo; i < hi; i++) a[i] = b[i] + SCALAR * c[i]; break; /* Triad */ + } + return NULL; +} + +static double now_s(void) { + struct timespec t; + clock_gettime(CLOCK_MONOTONIC, &t); + return (double)t.tv_sec + (double)t.tv_nsec * 1e-9; +} + +/* Returns GB/s for one pass of the given kernel across all threads. */ +static double bench(int kernel) { + pthread_t th[256]; + work_t w[256]; + size_t chunk = N / (size_t)NTHREADS; + double t0 = now_s(); + for (int i = 0; i < NTHREADS; i++) { + w[i].lo = (size_t)i * chunk; + w[i].hi = (i == NTHREADS - 1) ? N : (size_t)(i + 1) * chunk; + w[i].kernel = kernel; + if (pthread_create(&th[i], NULL, run, &w[i]) != 0) { + fprintf(stderr, "pthread_create failed\n"); + exit(4); + } + } + for (int i = 0; i < NTHREADS; i++) pthread_join(th[i], NULL); + double dt = now_s() - t0; + /* Copy/Scale touch 2 arrays (1 read + 1 write); Add/Triad touch 3. */ + double arrays = (kernel < 2) ? 2.0 : 3.0; + double bytes = arrays * (double)N * (double)sizeof(double); + return dt > 0.0 ? bytes / dt / 1e9 : 0.0; +} + +int main(int argc, char **argv) { + if (argc < 4) { + fprintf(stderr, "usage: %s \n", argv[0]); + return 2; + } + size_t array_bytes = strtoull(argv[1], NULL, 10); + NTHREADS = atoi(argv[2]); + if (NTHREADS < 1) NTHREADS = 1; + if (NTHREADS > 256) NTHREADS = 256; + int reps = atoi(argv[3]); + if (reps < 1) reps = 1; + + N = array_bytes / sizeof(double); + if (N < (size_t)NTHREADS) { + fprintf(stderr, "array too small for thread count\n"); + return 2; + } + a = malloc(N * sizeof(double)); + b = malloc(N * sizeof(double)); + c = malloc(N * sizeof(double)); + if (!a || !b || !c) { + fprintf(stderr, "allocation failed\n"); + return 3; + } + for (size_t i = 0; i < N; i++) { a[i] = 1.0; b[i] = 2.0; c[i] = 0.0; } + + bench(3); /* warm: fault pages in, prime, discard */ + + const char *names[4] = {"copy", "scale", "add", "triad"}; + printf("{\"threads\":%d,\"bytes_per_array\":%zu,\"reps\":%d,\"results\":{", + NTHREADS, N * sizeof(double), reps); + for (int k = 0; k < 4; k++) { + printf("%s\"%s\":[", k ? "," : "", names[k]); + for (int r = 0; r < reps; r++) { + printf("%s%.2f", r ? "," : "", bench(k)); + } + printf("]"); + } + printf("}}\n"); + + free(a); free(b); free(c); + return 0; +} diff --git a/examples/m5-2026/Issues/Performance Variance.md b/examples/m5-2026/Issues/Performance Variance.md index 3a4e0f1..0b870be 100644 --- a/examples/m5-2026/Issues/Performance Variance.md +++ b/examples/m5-2026/Issues/Performance Variance.md @@ -36,7 +36,7 @@ Recommended approach (script must do this): 1. Run Geekbench 6 multi-core **at least 5 consecutive times** with ~30s gaps. 2. Compute: max, min, mean, standard deviation, **(max-min)/mean × 100**. 3. Flag the unit if the spread exceeds ~10%. Healthy units are well under this. -4. Cross-reference each run's score against the published M5 Max baseline (~25k–28k Geekbench 6 multi-core, depending on chip variant). A unit consistently 20%+ below baseline is suspect even if variance is low. +4. Cross-reference each run's score against the published M5 Max baseline (~29,000–29,400 Geekbench 6 multi-core, ~4,200–4,335 single-core; see [Benchmark Reference](../../../Verification/Benchmark%20Reference.md) and verify live since numbers drift). A unit consistently 20%+ below baseline is suspect even if variance is low. 5. Repeat with a sustained workload (Cinebench R24 multi-core 10-min run) and watch for score drop between iterations. Companion checks during the runs: