Skip to content

Cleanup master plan: full fork assessment — slop removal, prize-research isolation, and the road to an upstream PR #499

Description

@lalalune

Full audit of lalalune/ArkLib (fork/main e66366b36+) vs upstream Verified-zkEVM/ArkLib (merge-base d61905b25, 2026-07-07; upstream only 5 trivial commits ahead). Fork is 14,415 commits / ~12,170 tracked files / +2.94M lines over upstream. This is the complete plan for what to delete, move, finish, consolidate, and eventually PR upstream. The upstream PR is Phase 4 — not yet.

Provenance: five parallel deep audits + a dedicated cruft sweep, 2026-07-09. Counts are measured (git diff --name-only d61905b25..fork/main aggregations, git ls-tree -r -l fork/main, tip-tree greps).


0. TL;DR

  • 91% of the delta is the δ prize campaign*: 4,259 files under ProximityGap/ (upstream: 23), 2,341 of them Frontier/ rungs, plus 30 MB scripts/probes and a 1,173-file kb.
  • Verdict on naming: ProximityGap/ is legitimately named — keep the name, evict the campaign. Upstream defines the directory as proximity-gaps literature formalization (BCIKS20/DG25/AHIV22) + protocol-facing abstractions (Basic, Folding, MCAGenerator, ProximityGenerators) consumed by BatchedFri/Stir/Whir. That is what it should contain — nothing else.
  • The campaign is a research program, not library content. Canonical home: top-level Research/ProximityPrize/ Lean lib target on branch research/proximity-prize; end-state a separate repo (lalalune/rs-proximity-gaps already exists) via lake require once churn stabilizes. Invariant: Research imports ArkLib, never the reverse — this single rule makes the upstream PR fall out for free.
  • The "core API blocker" dissolves on inspection: Lattice2, GSFactorExtract, CapacityBounds have zero real protocol consumers (their importers are campaign files); Hab25ConjectureGlue's only ProofSystem consumers are Whir/MCAConjecture* files that are themselves campaign work squatting in ProofSystem/. Only Errors.lean (17 external importers) has genuine library character.
  • Honesty flag: the tree is sorry-free, but carries 9 named axiom declarations and 151 *Residual open-hypothesis declarations across 82 files. "Done" was partly achieved by converting holes into named axioms/hypotheses — that is the real remaining proof debt and must never be silently discharged.
  • The upstream-worthy core: Binius/BinaryBasefold/FRIBinius/RingSwitching Binius closeout: remove residual hypotheses and restore focused build validation #313 closeout (sorry-free, needs consolidation), ~8 general ToMathlib lemma files (of 354!), ~10 modified shared-Data files incl. a genuine Fin.castSum correctness bug-fix.

1. DELETE — pure cruft, zero knowledge loss (git history retains all)

1a. Scratch & experiment mass

Target Size/count
scripts/probes/** 4,724 files, ~30 MB (3,893 .py; incl. .err/.out/empty files); zero CI references
scripts/{rust-pg,wf,cuda-pg,conjectures,deltastar-grind} ~89 files, ~710 KB; CI-unwired
Root-level scratch 205 files: _iNNN.json, _357_*.md/.patch, _ll_*.lean, _tmp_*.lean, _my_equiv_backup.lean, _ent2.txt(empty), _i357_* mega-notes (0.5 MB each)
Root duplicate insight-logs DISPROOF_LOG.md (1.1 MB), _disproof_log.md (0.7 MB), _dl_main.md (0.5 MB), _dossier_local.md — all stale copies of the canonical PG ledger
Root project-log strays AUDIT_LEDGER.md, journal.md, PAPERS_NEEDED.md, hd_peak.py, audit.sh
audit/ dir 3 files, stale 2026-06-10 snapshot

1b. Cruft sweep finds (new)

  • 10 committed arXiv/paper PDFsdocs/references/proximity-gap-paley-spectrum/*.pdf, ~6.4 MB of copyrighted papers. Replace with citation entries in kb/sources (which already exists for exactly this).
  • Committed GitHub-issue archaeology: docs/kb/audits/issue357-comments-archive{,-final}.json (~2.2 MB), _357_comments_all.jsonl (0.6 MB), _issue232_*, _issue304_*.
  • macOS AppleDouble file: ._deepband_extremal.md.
  • Mojibake Windows-path filename: C:UsersAdministratorarklib_a03_comments.json (0.7 MB).
  • Stray artifacts in the Lean tree: ToMathlib/SpartanBricks.lean.bak, OracleReduction/FiatShamir/StateRestorationTransport.lean.patch, Binius/.../Steps/FinalSumcheckExtractorDraft.wip, Binius/BinaryBasefold/docs/reversal-cluster-diagnosis-2026-06-24.md, ProximityGap/_444_issue_comment_draft.md, Frontier/Template.lean (scaffold template).
  • ~20 empty files (probe _out/_err.txt and root _ent* leftovers).
  • scripts/cuda-pg/results-growthlaw-*/rho4.out and similar result dumps.

1c. Repo bloat & branches

Est. Phase-1 shrink: ~75 MB and ~5,300 files, no proof surface touched.


2. RESTORE ProximityGap/ + CREATE Research/ProximityPrize/

2a. ProximityGap/ on main returns to canonical shape

  1. Upstream's 23-file skeleton (our modified Basic, BCIKS20/Curves diff-reviewed against upstream's).
  2. Our genuine extensions of the same papers, in upstream's paper-keyed subdirectory convention: the real parts of our 145-file BCIKS20/ subtree (ListDecoding, ErrorBound, WeightedAgreement — the latter gated on §3.1), real DG25/ additions.
  3. Errors.lean (ε-accounting API) beside Basic.
    No new names for library content — upstream's names ARE the names (no ProximityGap/Core/).

2b. Everything else → Research/ProximityPrize/ (branch research/proximity-prize)

  • Own lake lib target; PROXIMITY_PRIZE_WORKBENCH.lean is its root module (it already aggregates the machine-checked spine).
  • Moves: all 2,341 Frontier/ rungs (incl. the 0.6 MB CampaignProvenIndex.lean import index), 149 _probe/_wf/_scratch files, the 39-file QRWeil* adapter family, 105 _R1xx rungs, one-off numeric instances (F17×9, Fermat257×4, 65537×2, F41×2), 16 ALL-CAPS narrative .lean files, ~1,700 top-level research files, Lattice2, GSFactorExtract, CapacityBounds, Hab25ConjectureGlue.
  • Campaign tendrils outside PG move too: ProofSystem/Whir/{Hab25WindowMCA, MCAConjectureEllaryReduction, MCAConjecturePairReduction}; the ~300 paper/prize-named ToMathlib/ files (BCIKS20*, BKR06*, Beta*, Bridge2*, CS25*, CZ25*, AGL2x, Gauss/Paley/Wick/Saddle/Keystone); campaign bridges in GuruswamiSudan/ (real GS machinery stays); prize-support Data/Polynomial infra (RationalFunctionsCore +3027 etc.) travels with its consumers.
  • Knowledge corpus lives beside the code: Research/ProximityPrize/{DISPROOF_LOG.md (2.78 MB, 482 refutation certificates — the irreplaceable anti-repeat ledger), DOSSIER.md (v3 content, unversioned filename — git is the version), kb/papers/, kb/sources/, AGENTS.md}. Zero .md under ArkLib/** (currently 45–48). The nested duplicate kb (ProximityGap/docs/kb/, ~26 files) folds in or dies.
  • The 862 datestamped kb session notes + 16 round-notes + mega-notes (297 KB deltastar-100-attacks/README, 447 KB refute-round-25-lessons) drop — conclusions are distilled into LOG+dossier; history retains them.
  • Strict gates (laundering, zero-hole census, axiom audit) run over BOTH lib targets — the research branch keeps the honesty contract.
  • website/{deltastar-paper,truthmining} + their Cloudflare workflows accompany the research branch (they deploy; keep only if wanted).
  • Rule: Research → ArkLib imports only, never reverse.

3. FINISH / CLOSE OUT (real work)

  1. 9 named axioms: BCIKS20 weighted_correlated_agreement_* ×4 + large_agreement_set_on_curve_implies_correlated_agreement['] ×2 (= the paper's weighted Theorem 1.2 — real formalization project), Steps/Fold.lean foldKnowledgeStateFunction_toFun_full + foldOracleVerifier_rbrKnowledgeSoundness, CyclotomicRing/Subfield/Field.lean no_selfReciprocal_factor. Prove, or register as documented external assumptions with issue refs.
  2. 151 *Residual declarations / 82 files: triage — honest campaign-conditionals (move with Research), Binius grant-out-of-scope (documented in docs/wiki/Binius_Closeout_Audit.md, fine), plain unfinished (finish).
  3. DuplexSponge *Paper cluster (15 files, OracleReduction/FiatShamir/DuplexSponge/Security/): structurally incomplete; finish before OracleReduction upstreaming.
  4. Binius WIP hotspot: BinaryBasefold/Relations.lean (20 TODO-family markers), Basic.lean (8), Steps/Fold.lean (7) — resolve or convert to documented boundaries.
  5. Prove δ* — complete research dossier (v3) for the RS proximity-gap prize (successor to #464) #466 stays open — it IS the research; the honesty contract forbids closing it without the wall.

4. CONSOLIDATE / REFACTOR

  • One-lemma-per-file explosion: Binius Soundness/ (30+ micro-files), OracleReduction/Composition/Sequential/Append* (~40 micro-files) — merge into upstream-conventional file counts.
  • ToMathlib split: keep ~8 general candidates (BalogSzemerediGowers, SumProduct, AppendHelpers, AveragingExistence, BivariateDegreeToolkit, BivariateGradedDvd, CoeffExtract, CyclotomicPatternInjectivity); everything paper-named moves (§2b). Delete ToMathlib/README.md if stale.
  • ArkLibScratch.Issue62 namespace in ProofSystem/BCS/ErrorAccounting.lean — rename; scratch namespaces can't ship.
  • Adopt upstream's CommitmentScheme/ retirement: fork still tracks the legacy dir (4 files) upstream deleted — migrate remains into Commitments/Functional/, delete.
  • Comment slop pass over the kept surface: change-narration, session-log comments, campaign archaeology in docstrings — headers describe the math, not the hunt.
  • AGL24* (119 files), GMMDS (50), GuruswamiSudan (44), ListDecoding (45) under CodingTheory: same research-drift flavor — triage each family lib-vs-Research with the §2b rule.

5. UPSTREAM PR CARVE-OUT (Phase 4 — later)

  1. Rebase over origin/main (5 commits: Rq lemma move refactor: move constRq and Rq degree/coef lemmas into Rq.lean Verified-zkEVM/ArkLib#621 + CI tweaks — trivial).
  2. Dependency re-pointing (the hard blocker): lalalune/VCV-ioVerified-zkEVM/VCV-io v4.30.0; lalalune/CompPolyVerified-zkEVM/CompPoly v4.30.0; fork-only PolyFun dep upstreamed-or-dropped; doc-gen4 v4.29.0 → v4.30.0; regen manifest. (Note: this pinning is also why the auto-update workflow can never succeed — see Updates available but manual intervention required #468 diagnosis.)
  3. Exclude fork-only workflows: deltastar-cloudflare, truthmining-cloudflare, sorry-tracker.
  4. PR contents: consolidated Binius/RingSwitching closeout; the 8 general ToMathlib picks; reviewed shared-Data improvements (Fin/Basic castSum bug-fix + induction_three/four, Fin/Sigma, Fin/Tuple/Lemmas cleanup, Matrix/Vandermonde, MvPolynomial/Multilinear, CodingTheory/Basic/{Distance,LinearCode}, Data/Probability/Instances, Polynomial/SplitFold after review); BCIKS20/DG25 paper-extension files from §2a. The PR then reads as "extend BCIKS20/DG25 formalizations + Binius closeout + shared fixes" — reviewable.

6. EXECUTION ORDER

  • Phase 1 — mechanical (no proof risk): all of §1; ~75 MB / ~5,300 files gone; gitignore _generated/{declarations.json,dedup-report.md}; prune branches; kb dedup.
  • Phase 2 — structural: restore ProximityGap/ canonical shape; create Research/ProximityPrize lib + research/proximity-prize branch; move the campaign + tendrils; CI green on both.
  • Phase 3 — quality: §4 consolidations, §3 axiom/Residual triage, comment pass.
  • Phase 4 — upstream: §5. Not yet.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions