[Tool] parameter-golf-checker: static analysis reviewer aid for submission triage

[Bortlesboat]

With 1,500+ submissions now in the queue, I built a small static-analysis tool that might help narrow down review workloads: parameter-golf-checker.

Repo: https://github.com/Bortlesboat/parameter-golf-checker

What it is

A pure-Python AST-based checker that reads a submission's train_gpt.py and surfaces patterns that commonly correspond to compliance issues from #1017. No GPU, no execution — it just parses the file and walks the tree. It handles the lzma+base85 compressed wrapper shape used in several recent records.

What it checks (8 checks total)

• C1 Causality, C2 Normalization, C3 Score-before-update, C4 Single-pass
• N-gram cache / oracle-selection patterns
• SLOT / LBFGS / torch.linalg.solve / second-order updates on val
• 16 MB artifact estimate
• Network imports

What it is NOT

It is explicitly a reviewer aid, not an authoritative PASS/FAIL judge. Every PASS is a weak signal, every WARN means "human should look at this," and every FAIL is a hypothesis. Aliasing, helper-indirected scoring, and metaprogramming can all fool it. I tightened it after a harsh code review until it emits zero false FAILs on the 10 most recent merged records I tested (including #1555, #1560, #1561, #1572, #1576), but I'm sure there are still patterns it misses in both directions.

Usage

```bash
python pgolf_checker.py --pr 1576
python pgolf_checker.py --file path/to/train_gpt.py
python pgolf_checker.py --pr 1576 --format json
```

MIT licensed. Happy to fold in whatever the organizers find useful — if there's a violation pattern I'm missing or a check you'd want added, issues and PRs welcome. And if you'd rather it not exist, just say so and I'll take it down.

[Bortlesboat]

Quick follow-up. I swept the tool across 30 open Record-claiming PRs and the results are committed under sweep_results/.

Aggregate (30 Record PRs)

• 0 hard PASS — tool is intentionally conservative, PASS requires unambiguous evidence
• ~23 WARN — mostly "pattern looks legal but I can't verify across helper functions"
• 6 C3 FAIL — all six sit in ttt_adapt_adamw / prequant_ttt_adapt_adamw style helpers that take val tokens directly and run optimizer.step() with no preceding no_grad scoring block in the same function scope
• 1 parse failure (Record: Asymmetric Two-Lane Parallel Routing + Tap-In V6 + Legal TTT (1.073938) #1518, unicode em-dash in a comment)

Most of the 6 C3 FAILs are likely false positives from helper-indirected scoring, but I manually traced one (#1488) and it looks like a real score-after-update pattern — posting line-level evidence on that PR separately.

Retrospective validation against closed PRs

• [RECORD] L-BFGS SLOT + Entropy-Adaptive N-gram Mixer (0.2282 BPB) #1507 (L-BFGS SLOT + N-gram mixer, 0.2282 BPB, closed): tool returns FAIL on C3 with TTT loop at line 909 in eval_val_slot directly references val data at lines 919, 920 with optimizer.step() at line 950 and no preceding no_grad scoring block. Also flags LBFGS at line 939.
• Score-First TTT + Causal N-gram (order=82) — val_bpb 0.29882 (3-seed mean) #1605 (Score-First TTT + Causal N-gram order=82, 0.29882 BPB, closed): multi-WARN with hessian second-order hints + n-gram cache + val references.

Both were closed by maintainers, and the tool's output lines up with the likely reasons — retrospective fingerprints on already-decided cases.

Known limitations I'm fixing

1. C1 Causality broadened to recognize flash_attn_* calls with is_causal=True, not just scaled_dot_product_attention.
2. Helper-indirected scoring — the tool can't currently follow eval_val_ttt(model, val_tokens) → helper that does with torch.no_grad(): score(); ttt_adapt(). This is the source of most C3 false positives.
3. Unicode handling in the fetch path.

Offering

If this is useful to the organizers I'm happy to:

1. Run the sweep on new submission batches and post triage reports
2. Add more checks as new violation patterns appear
3. Wire it up as a GitHub Action that auto-comments on new record PRs, whichever integration point you'd prefer

Tool: https://github.com/Bortlesboat/parameter-golf-checker
Raw sweep outputs: https://github.com/Bortlesboat/parameter-golf-checker/tree/main/sweep_results