shakedown: wave 8: in-store verification toolkit (--store, compare-reports, STREAM triad, opt-in --llama)#5
Merged
Merged
Conversation
…ase 12), opt-in --llama (phase 14) --store turns on --noaccel + --gpu with a longer warmup and more iterations for verifying a new unit. compare-reports.sh diffs a unit against a known-good sibling (the calibration-free signal). Phase 12 now prefers a vendored STREAM triad (stream-triad.c, compiled at runtime with clang, no network) and falls back to the memmove proxy. Opt-in Phase 14 (--llama) clones+builds llama.cpp and runs llama-bench (combined CPU+GPU+memory load); off by default, skips cleanly without git/cmake/network/model, optional LLAMA_MODEL_SHA256 integrity check. lint.yml REQUIRED_PHASES adds 14_llama_bench.
…res, docs Verification/Benchmark Reference.md: sourced Cinebench/Geekbench baselines per generation, in-store + hotel protocol, HK return constraint, mianibench + Geekbench live-lookup. Schema 1.3 (phase 12 triad fields/method, phase 14_llama_bench). SECURITY.md discloses the default-run pure/offline property plus the two opt-in exceptions (--gpu compile, --llama network/clone/build) and the model-download integrity caveat. Corrected the stale M5 Max baseline (~25-28k to ~29,000-29,400 GB6 multi). README/Runbook/CONTRIBUTING/CHANGELOG wiring.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The in-store toolkit for verifying a new unit, plus the open-source-test integration.
New
--storethorough profile: turns on--noaccel+--gpuwith a longer warmup and more iterations, so an intermittent batch defect has more chances to surface.compare-reports.sh REFERENCE.json UNIT.json: side-by-side diff that flags where a unit trails a known-good sibling. With the absolute thresholds still uncalibrated, this unit-to-unit comparison is the most trustworthy in-store signal.stream-triad.c(real copy / scale / add / triad), compiled at runtime with clang, no network. Falls back to the pure-Pythonmemmoveproxy when clang is absent. Closes the copy-only-proxy gap;details.methodrecords which ran.--llama(opt-in): clones + builds llama.cpp at a pinned ref and runsllama-bench, a combined CPU+GPU+memory AI load (the workload class the M5 defect was reported on). Off by default; reaches the network and runs third-party code (disclosed in SECURITY.md); skips cleanly without git / cmake / network / model; optionalLLAMA_MODEL_SHA256integrity check.Verification/Benchmark Reference.md): install-this / run-this / expected-score guide with sourced, adversarially-verified Cinebench R24 + Geekbench 6 baselines per generation (M1-M5 families + Intel 2019), the in-store and hotel protocols, the Hong Kong return constraint, and live-lookup links (Geekbench Browser, mianibench.com crowd distributions). Corrected the stale M5 Max baseline (~25-28k to ~29,000-29,400 GB6 multi-core).Open-source-test design
Two patterns, picked to keep the default run honest: vendor a single-file source compiled at runtime with a pure fallback (the STREAM triad, no network), and a strictly opt-in phase that may use the network or third-party code (
--llama), off by default and disclosed in SECURITY.md. The default./runstays pure bash + Python stdlib and offline.Schema + CI
Schema 1.3 (Phase 12 triad fields + method; new
14_llama_benchkey, a skipped placeholder by default). CI REQUIRED_PHASES updated. Backward compatible.Verification
Validated on an Intel MacBook Pro 16 (2019). The STREAM triad runs and produces real numbers here; the default and all-flags paths both produce schema-1.3 reports with all 16 phases. The
--gpuand--llamahappy paths run on a Metal / cmake-capable machine; their skip paths are verified here. Adversarial review came back clean on correctness, bash portability, and voice; the one minor finding (model-download integrity) is fixed.