Skip to content

judge: callable relation models and configurable training pipeline - #5

Draft
karthikreddy-02 wants to merge 3 commits into
mainfrom
judge-finetune
Draft

karthikreddy-02 wants to merge 3 commits into
mainfrom
judge-finetune

Conversation

@karthikreddy-02

@karthikreddy-02 karthikreddy-02 commented Aug 9, 2026 •

Copy link
Copy Markdown
Collaborator

Equation-pair scoring and judge training need reusable Python interfaces for notebooks and representation experiments. This PR adds the trained-judge library, CLI, training scripts, and a configuration-driven pipeline.

Load a Judge once, then call judge(a, b) or compare_many(pairs). Verdicts contain the relation, normalized label scores, and original equations. The loaded PEFT model remains accessible for activation hooks. Labels describe B relative to A.

The pipeline exposes run(config, stage=..., dry_run=...) and an equivalent CLI. Configuration controls rendering, prompts, LoRA settings, checkpoint retention, and evaluation metrics. It supports infix, Lean, LaTeX, Python lambda, and function-call equation notation.

Implementation details:

  • Package imports work from the repository root. Data generation uses the checkout's oracle directory by default.
  • Prompt metadata and tokenizers accompany intermediate checkpoints and the final adapter.
  • Evaluation records selected metrics with explicit denominators and propagates subprocess failures.
  • Data reuse requires a matching recorded recipe. Training protects existing final adapters and their configuration.
  • Historical results remain available with explicit qualifications. Normalized scores are not calibrated correctness guarantees.

Validation:

  • 20 offline unit tests passed, including synthetic scoring, padding, batching, rendering controls, metadata, and failure propagation.
  • Infix and Lean pipeline dry runs passed.
  • Module and script CLI smoke checks, compilation checks, and git diff --check passed.
  • An offline GitHub Actions workflow runs the tests and dry runs.

Limitations:

  • Real Gemma/Llama inference and a GPU training smoke test remain outstanding. Tests do not download base weights.
  • The public pipeline uses the existing label-balanced sampler. Local shape-balanced v2/v3 work and shape diagnostics remain separate.
  • The existing sampler permits repeated underlying pairs. Class-held-out evaluation requires at least one novel endpoint, not necessarily both.
  • These judges predict relation tokens; they do not generate chain-of-thought text. This PR establishes no new causal interpretability result.
  • Dataset reuse checks configuration, not the identity of the oracle files. Changed oracle artifacts require a fresh data directory.

Generated data and model weights are excluded. Keep this PR in draft until model smoke tests and review are complete.

Replaces the prompted-LLM judge with a trained artifact: known training data,
a known split, and weights we own and can probe. Labels come from oracle/'s
Lean-verified implication matrix, so the training signal has zero label noise
and no prover runs at training time.

The split is by equivalence class, never by pair (1,064/140/211 of 1,415).
Laws inside a class are mutually derivable, so a random pair split lets a
lookup table score perfectly. The x = y class alone is 31.9% of the catalogue
and is pinned into training. pairs_val minus classes_val is therefore the
memorization measurement: 1.91 pp on Gemma-2-2B, 1.25 pp on Llama-3.1-8B.
Both trained a judge, not a lookup table.

judge.py exposes both a CLI (compare / batch / rank) and a Judge class, with
the live PeftModel reachable as .model so the interpretability work can attach
hooks rather than shell out. Scoring restricts logits to the four label tokens
before the argmax, so an invalid answer is impossible by construction rather
than merely unlikely.

Results and ablations across two model families are in judge/README.md,
including two findings that did NOT survive replication and should not be
claimed: the attention/MLP asymmetry is a Gemma artifact, and the
early-vs-middle layer ordering flips between families. What does replicate:
rank saturates at or below 16, and late-layer-only adaptation collapses
(5-6x worse than any other layer choice) in both models.

Neither base model can be contaminated: ETP launched 2024-09-25, while
Gemma-2 shipped 2024-06-27 and Llama-3.1 on 2024-07-23.

Generated data and trained weights are gitignored; both are regenerable.
@karthikreddy-02 karthikreddy-02 changed the title judge: trained ETP relation judge — library, CLI, training and eval judge: callable relation models and configurable training pipeline Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant