Skip to content

Audit original structural compression on 1.85M trained n-gram rows - #42

Merged
initial-d merged 10 commits into
mainfrom
codex/large-structural-audit
Oct 31, 2026
Merged

initial-d merged 10 commits into
mainfrom
codex/large-structural-audit

Conversation

@initial-d

@initial-d initial-d commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Large n-gram tables can share training updates while their frozen vectors remain different. Add a CPU workflow that audits actual keyed rows, then measures the original DATrie/mark-compact contribution against raw, native identity, lexical layout and exact encoded-dedup controls at the same encoding.

On the unchanged 1,855,907 × 256 finite SGNS table, conditional identical-word-support updates are almost collinear, but final-vector cosine is only 0.21–0.35. Complete FP32/FP16 structural savings are 0.12–0.63%; INT8 remains larger than raw INT8, and broad native query misses remain slower. Quantization/joint sharing are explicitly lossy; all encoded representative bytes and original IDs are independently verified. This is storage/vector evidence, not LLM task-quality or convergence evidence.

  • 27 fully exported layouts, every original ID verified, and 108 fresh query workers across two randomized rounds per trace. No GPU use or retraining.
  • Actual initialization and original training-ID provenance explain the co-training signal; no synthetic vectors are used for compression evidence.
  • Complete raw reports, four figures, source/environment fingerprints, retained launcher failure and partial-controller handoff, checks and reproduction commands: evidence report.
  • Validation: 16 current targeted tests pass; Linux full-suite snapshot has 145 tests/17 skips, local snapshot 145/26 skips. Final-head CI at e5950af755238c7e0706089d1112e4cd3d699afb: 18/18 passed before merge. Fabricated correctness fixtures remain separate from measured data.

@initial-d
initial-d merged commit 33dd75e into main Oct 31, 2026
18 checks passed
@initial-d
initial-d deleted the codex/large-structural-audit branch October 31, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant