docs: reconcile arXiv benchmark figures after post-fix review

[cdeust]

Closes #347

What changed

• Align the README LoCoMo headline with the canonical post-fix run: Recall@10 94.35%, MRR 0.8279, n=1,986, code SHA 2f45bcb.
• Add a supersession warning to the pre-fix results document.
• Resolve the provenance of the historical 0.794 / 0.926 comparator and label it explicitly as noncanonical.
• Align the current LongMemEval headline on the clean July run: Recall@10 98.2%, MRR 0.9167, n=500, code SHA 28145f0; retain 98.4% / 0.9124 only as the named historical E1 v3 ablation snapshot.
• Label BEAM 0.591 as a five-conversation, n=100 retrieval-proxy MRR at code SHA a071d89 and remove the invalid comparison with the paper end-to-end metric.
• Align the thermodynamic and context-assembly arXiv sources and all publication-facing appendices, then regenerate both PDFs.
• Add a versioned audit of the arXiv-facing figures and their sources.

Opus 5 review follow-up

Resolved the residual stale 92.6% in the Popper appendix, the LongMemEval 98.4 / 98.2 / 97.8 split, and the missing BEAM metric/protocol audit identified in the issue review.

Why

The public documents mixed current post-fix results with historical or protocol-specific measurements. This made the arXiv draft ambiguous and, for BEAM, risked comparing incommensurable retrieval-proxy and end-to-end metrics.

Validation

• scripts/check_doc_claims.py
• git diff --check
• Relative Markdown-link validation for all changed Markdown files
• Exact artifact/provenance assertions for LoCoMo and LongMemEval
• Exact 14-row LoCoMo table parity across the post-fix result, Markdown paper, and LaTeX paper
• Clean LaTeX builds for both papers, with no undefined citations or references
• PDF text extraction confirming 98.2%, 94.35%, and 0.591 with their metric framing

[cdeust]

ZETETIC-REVIEW: APPROVE

Reviewed the diff and the audit document it carries, not the title.

This corrects published figures, and it corrects them downward. LongMemEval R@10 goes 98.4% → 98.2%, LoCoMo temporal reasoning 0.926 → 0.917, single-session preference accuracy 93.3% → 90.0%. A documentation change that makes a paper's claims weaker is the opposite of the failure mode this kind of PR usually carries, and it is the reason to trust the rest of it.

Every figure is traced to an artefact, not to a memory. docs/benchmarks/arxiv-figure-audit-2026-08-02.md gives, per benchmark, the run record path, the code SHA, the dirty flag and the date — 28145f0b clean for the LongMemEval headline, 2f45bcb3 clean for the post-fix LoCoMo, a071d89 for the BEAM five-conversation protocol. A reader can re-run the claim instead of trusting the table.

The root cause of the drift is named: the first audit had promoted the May E1 v3 ablation baseline to the current headline while README carried a different number. Historical values are not deleted — they are retained under an explicit historical label with their own named run, which keeps the record honest rather than tidy.

The noise-floor correction is the strongest signal in this diff. Post-fix LoCoMo consolidation-only values sit inside the stated ±0.002 MRR floor, so HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE at +0.0017 are now described as positive-direction observations within noise rather than causal contributions. Retracting a causal claim you are entitled to keep, because the measurement does not support it, is the standard this project asks for.

Scope: documentation only — zero .py, .sh, .toml or .yaml files. CI 19 pass, 1 skipping.

No blocking findings.

[cdeust]

ZETETIC-REVIEW: REQUEST_CHANGES

This supersedes my earlier APPROVE on this PR, which was wrong.

I approved it because the figures moved downward, and I read that as intellectual honesty. It is the opposite of what the contract requires. Levelling down is accepting a regression, and a regression is fixed — never accepted, never annotated, never published as the new truth.

What this PR currently does:

figure	before	after
LoCoMo temporal reasoning	0.926	0.917
LoCoMo single-session (preference)	93.3%	90.0%
LoCoMo single-session (user)	—	95.7% (from 94.3%)
LongMemEval R@10 headline	98.4%	98.2%

Two different things are mixed here, and only one of them is legitimate:

1. The LongMemEval change is a provenance correction, not a regression: the audit shows 98.4% was the May E1 v3 ablation baseline wrongly promoted to headline, and 98.2% is the current clean run at code SHA 28145f0b. Correcting which run is authoritative is right, and the audit document that establishes it is good work — that part stands.

2. The LoCoMo temporal-reasoning and single-session-preference drops are a real degradation introduced by the fix, and this PR publishes them instead of removing them. Temporal reasoning losing 0.009 and preference accuracy losing 3.3 points is the fix costing capability somewhere. That cause is what has to be found and corrected; the paper should then carry the restored figures.

What has to happen before this lands: root-cause the post-fix degradation on those two dimensions and fix it, so the published figures are at least the pre-fix values. If, after the fix, a dimension genuinely cannot reach its previous number, that is a finding to bring to the owner with the measurement — not a line to edit in a paper.

The noise-floor reclassification (HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE at +0.0017 described as within-noise rather than causal) is correct and should be kept as-is: that is a claim being retracted for lack of support, which is a different act entirely from lowering a measured result.

Note also that this branch no longer applies cleanly to main and needs a rebase; the conflict in docs/papers/thermodynamic-memory-vs-flat-importance.md is semantic — the branch carries the corrected LoCoMo figure while main carries the repository rename to ai-architect-mcp-codebase, and both sides are right about their own half.

[cdeust]

Correction of my previous verdict's reasoning, on evidence I should have checked before writing it.

My REQUEST_CHANGES claimed the per-category drops were a degradation introduced by a fix. I verified that claim against the artefacts and it does not hold. Two errors on my side:

1. I misattributed the table. "Temporal reasoning", "Single-session (preference)" and the rest sit under LongMemEval in README.md, not LoCoMo. I cited them as LoCoMo figures. The two surfaces publish near-identical row labels, which is how I got there, but the mistake is mine.

2. There is no degradation — the high figures have no provenance. Same experimental condition on both sides (with_consolidation = false):

	authoritative run	source of the high figures
artefact	benchmarks/results/repro/20260714-v4.14.1-pretag/longmemeval-s.json	benchmarks/results/ablation/longmemeval-s_v3/BASELINE.json
repro	git_commit 28145f0b…, git_dirty false, lib versions, reranker model SHA-256	None
overall R@10	0.982	0.984
Temporal reasoning R@10	0.9774	0.9850
Single-session (preference) R@10	0.90	0.9333

The higher numbers come from a run that records no code SHA, no dirty flag and no environment — it cannot be attributed to a commit or reproduced. The lower numbers come from a fully traced clean run. That is the same defect class as the 98.4 → 98.2 headline already corrected in this PR: an ablation snapshot promoted to headline.

The test that separates the two cases: if the older figure came from a different run — different protocol, ablation, or code revision — it is provenance, and documentation settles it. If it came from the same protocol before a code change, it is a degradation caused by that change, and editing the paper publishes the regression. Here the manifests answer the question directly, and the answer is provenance.

So the levelling-down objection does not apply to these rows, and my REQUEST_CHANGES was wrong on its central factual claim. The principle behind it stands and is not softened: a regression is fixed, never published. It simply is not what is happening here.

What still blocks this PR is mechanical: the branch no longer applies to main. The conflict in docs/papers/thermodynamic-memory-vs-flat-importance.md is semantic — the branch carries the corrected LoCoMo figure while main carries the repository rename to ai-architect-mcp-codebase, and each side is right about its own half.

Note for whoever picks this up: PR #414 reworks the same ground with this distinction applied, and an independent review of its 30-run provenance analysis is in progress. The LoCoMo 0.794 / 0.926 comparator is a separate claim from the LongMemEval rows above and still needs its own verification.

[cdeust]

Second correction, and it reinstates the original objection. The degradation is real.

My previous comment claimed the high figures had no provenance, based on BASELINE.json carrying repro: None. That was an incomplete search, and it produced the wrong answer.

The provenance exists, one directory level up:

benchmarks/results/ablation/longmemeval-s_v3/manifest.json
  code_hash  = 0e858e8db0f8a5dae0879fa0134113d101be19f8
  dirty      = false
  started_at = 2026-05-02T22:39:22Z
  n          = 500
  with_consolidation = false

$ git cat-file -t 0e858e8db0f8a5dae0879fa0134113d101be19f8
commit
$ git log -1 --format='%h %ad %s' --date=short 0e858e8d
0e858e8d 2026-05-03 feat(verif): blend-weight calibration results …

The run-level repro block is null because this campaign records provenance per directory under code_hash, not per row. Same convention the rest of the repository uses. The commit is real and reachable.

So both sides are attributable, under the same condition (with_consolidation = false, n = 500):

	date	commit	overall R@10	Temporal	Preference
ablation campaign	2026-05-03	0e858e8d	0.984	0.9850	0.9333
current headline	2026-07-14	28145f0b	0.982	0.9774	0.90

That is a measured drop between two attributable commits under an identical protocol. By the test I stated — same protocol before and after a code change means degradation, different run means provenance — this is degradation, and publishing the lower values is publishing the regression.

The thirty-run argument cannot establish otherwise: those runs span 2026-07-08 to 2026-08-09, and the high value is from 2026-05-03. The interval that would contain the boundary, 3 May to 8 July, was never examined. Absence of a boundary was demonstrated in a window that could not contain it.

My original REQUEST_CHANGES stands. The path forward is to bisect between 0e858e8d and the first July run, find what cost 3.3 points of single-session preference accuracy and 0.8 of temporal reasoning, and fix it — then publish figures at least at the May level.

A separate finding that survives all of this: the LoCoMo headline this PR promotes, 94.35% / 0.8279, has no committed per-query artefact either. The rule "no artefact, no figure" is being applied to the old comparator and not to the replacement. Either commit the artefact that supports the new headline, publish the value the existing artefact supports, or state plainly that no LoCoMo figure is currently traceable. The third answer is acceptable if it is owned.

For the record, this is my third verdict on this PR: approve, then request-changes on a misattributed table, then a correction that itself rested on a one-field search. The conclusion is back where it started, but only the version with the manifests in it is worth anything.

[cdeust]

Remplacée par #414, mergée.

Cette PR nivelait les chiffres publiés vers le bas sans distinguer deux causes de nature opposée. Le critère qui les sépare : un ancien chiffre issu d'un run différent — autre protocole, ablation, autre révision — est une erreur de provenance, et la documentation se corrige ; un ancien chiffre issu du même protocole avant un changement de code est une dégradation causée par ce changement, et éditer le document revient à publier la régression.

#414 applique cette distinction. Sur LongMemEval, elle établit par les manifestes — 0e858e8 et 28145f0b, même harnais, n=500, with_consolidation=false, arbres propres des deux côtés — qu'il s'agit d'une régression établie sur deux catégories, et non d'un défaut de provenance. Le README retire ces deux lignes au lieu de publier les valeurs basses comme références ; seul le commit causal, dans une fenêtre de 269 commits vérifiée au commit près, reste à localiser.

Elle traite par ailleurs le second constat soulevé ici : l'en-tête LoCoMo 94,35 % / 0,8279 n'avait aucun artefact par-requête committé. La règle « pas d'artefact, pas de chiffre » y était appliquée contre l'ancien comparateur et pas contre le nouveau. #414 revient à ef178da7, seule paire adossée à un manifeste — MRR 0,8278244, R@10 0,9419778 — en signalant explicitement au lecteur qu'il s'agit d'un run antérieur au correctif de plasticité.

Merci pour l'objection maintenue avec la preuve : elle a évité qu'une régression soit publiée comme valeur de référence.