Gemma 4 12B SFT-uplift PoC (+ MTP) on NVIDIA RTX 4000 Ada#129
Draft
ulises-c wants to merge 134 commits into
Draft
Gemma 4 12B SFT-uplift PoC (+ MTP) on NVIDIA RTX 4000 Ada#129ulises-c wants to merge 134 commits into
ulises-c wants to merge 134 commits into
Conversation
- Patch caching_allocator_warmup on HIP/ROCm to silence "Found no NVIDIA
driver" raised by the CUDA-only allocator warmup in recent transformers
- Force device_map={"": 0} for live QLoRA quantization so bitsandbytes can
quantize all layers on GPU (auto can dispatch to CPU, which bnb rejects)
- Add Stage 2 SFT config for Gemma 4 26B-A4B-IT on R9700 (gfx1201, NF4)
Closes #100
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pivot the gfx1201 SFT path to the community pre-quantized checkpoint unsloth/gemma-4-31B-it-unsloth-bnb-4bit (issue #100): it downloads already-NF4 (~19 GB), skipping the ~62 GB BF16 CPU staging at load that was the R9700's actual blocker — no L40S prequant + rsync needed. - Add `make train-gemma4-31b-stage2-unsloth` (TRAIN_PREQ=true + unsloth base) - Extend device_map={"":0} to the whole QLoRA path: the unsloth checkpoint is dynamic 4-bit (embeddings/lm_head/gates stay 16-bit) and "auto" could dispatch one of those modules to CPU, which bitsandbytes rejects - Document the prequant vs live-quant launch paths in the 31B config - Drop the 26B-A4B config (wrong target; 31B is the real one) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… fix Training data (dataset.py): - Add socrat-zh-sft / socrat-en-sft sources: one (system, user, asst) triple per dialogue turn using the exact message format socrates_teacher() sends at inference (历史对话记录 blob + 当前学生输入 label + 7-line rules system prompt). Fixes the train/serve schema drift that caused output collapse on the 5090 run (repetition loop, never emits EOS, ~45 min/call to context limit). - Split at dialogue level before per-turn expansion so all turns of a dialogue stay in the same train/test partition — prevents history blobs in test records from containing teacher responses seen during training. - Render-diff verified MATCH: True (system and user byte-identical to inference). - Existing socrat-zh / socrat-en multi-turn loaders untouched (backward compat). train_sft.py: - Only pass quantization_config to from_pretrained when not None. Explicit None caused auto_factory to overwrite the unsloth checkpoint's config.json quantization_config via from_dict(**unused_kwargs), crashing supports_quant_method. - W&B: auto-detect via _check_wandb() (wandb.login relogin=False); warn and disable if not authenticated. report_to=["wandb"] wired into SFTConfig with run_name defaulting to output-dir basename. pyproject.toml: wandb>=0.18.0 added as core dependency. .gitignore: wandb/ local run artifacts added. TRAIN_SOURCES updated to socrat-zh-sft,socrat-en-sft in stage2 config. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add tests/test_sft_inference_format.py: unit tests for the per-turn socrat-zh-sft / socrat-en-sft sources plus a render-diff gate that drives the real socrates_teacher() path (intercepting call_teacher_wrapped) and asserts the SFT record renders identically under the chat template's per-message trim. Pins the train/serve schema-drift fix (#94, #101) so future drift breaks CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-no-sync
This venv has the packages importable but not their console-script entry
points, so `uv run --no-sync pyright|codespell|pytest` failed to spawn
("No such file or directory"). Switch the type-check, spell-check, and test
steps to module invocations (`python -m pyright` / `codespell_lib` / `pytest`),
which resolve without the console scripts and without forcing a sync. Run
`make install-hooks` to refresh .git/hooks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…te runbook) Captures current state (structural drift fixed in d04bc83; test gate in e5a8a7d), the two remaining residuals to fix on this dev box (action line via get_action_for_state, eval line via the dataset's free-form evaluation field), the test-gate updates, and the R9700 runbook (faithful free-form-eval EOS gate before the full run). Bilingual (#108) explicitly deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The committed socrat-zh-sft / socrat-en-sft loaders still drifted from the real socrates_teacher() inference prompt on two user-message lines, the last residuals of the #94 train/serve schema mismatch that collapsed the first Stage-2b model into a non-terminating repetition loop: - action line: emitted the raw dataset turn["action"]; inference sends get_action_for_state(state). Systematic mismatch (e.g. a1: "生成一个问题" vs "生成一个与解题相关的子问题") on every opening teaching turn. - eval line: emitted templated "学生处于 {state} 状态"; the dataset already carries a per-turn free-form `evaluation` field, far closer to the live consultant prose inference actually sends. Changes: - socratic_teaching_system.py: hoist state_to_action to a module-level STATE_TO_ACTION + get_action_for_state() so the loader shares the exact inference source of truth (behavior-preserving; AST-verified identical to HEAD; live socrates_teacher render byte-for-byte unchanged — no eval re-baseline). - dataset.py: _build_inference_user_message now takes `evaluation`; both -sft builders use get_action_for_state(state) for the action line and the dataset `evaluation` field for the eval line (fallback to the old template only if absent). Target-turn gating still keyed on dataset state+action so the record set is unchanged. - test_sft_inference_format.py: un-rig the action dimension (parity test now asserts the action line render-equal too) and isolate the eval line as the sole, expected residual under a free-form consultant. Verified on this dev box: full suite 154 passed / 2 skipped; the real socrat-zh-sft / socrat-en-sft sets (42,890 records each) carry 0% template- fallback evals; official dry-run plumbing passes. Refs #101, #94. Bilingual EN/ZH (#108) deferred — EN stays Chinese-scaffolded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up PR #99 (Qwen3.5 training, flash-linear-attention CUDA guard, AMD/NVIDIA split GPU smoke tests). .env.example conflict resolved by merging both WANDB comment blocks; uv.lock takes main's version. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove expandable_segments:True from all PYTORCH_HIP_ALLOC_CONF entries; HIP allocator does not support it and the warning preceded GPU page faults at step 16 on gfx1201 - Switch eos-gate target to TORCH_USE_HIPBLASLT=1 — Gemma 4 uses standard softmax attention (no linear-attn delta-rule kernel), so hipBLASLt is safe and avoids the rocBLAS fallback overhead - Replace nohup with setsid on eos-gate launch so the whole process tree is in its own group and can be killed cleanly without leaving GPU dirty - Add docs/HANDOFF_SFT_SCHEMA_DRIFT_FIX.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Consolidated, externally-shareable writeup of the gfx1201 training findings from #100/#79/#101 — framed against the NVIDIA/CUDA baseline, with a confidence taxonomy (confirmed/untried/upstream/correction). Companion to the serving-focused GPU_SUPPORT.md. Tracks #109. Also allow "hsa" in codespell (HSA_OVERRIDE_GFX_VERSION env var). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the ulises-c/csen-346 repo link at the top and in references, and make GPU_SUPPORT.md links absolute so they resolve in the public gist (relative links break in the flat gist copy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Commit 860fc5f (this branch, 2026-05-30) stripped expandable_segments:True from all PYTORCH_HIP_ALLOC_CONF entries after its "unsupported" warning preceded GPU page faults at step 16 — reversing the PR #79 "works and helps" finding the report had presented as settled. Reframe as contested with the full three-flip timeline; latest empirical verdict is OFF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wandb>=0.18.0 was promoted to a core dep in pyproject.toml but uv.lock still listed it only under the wandb extra, failing CI's `uv lock --check`. Regenerated; no version changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ndoff transformers 5.9 apply_chat_template(..., return_tensors="pt") returns a BatchEncoding dict, not a raw tensor; extract input_ids before passing to model.generate. Also skip outputs/ in codespell (tokenizer.json vocab fragments are not real words). EOS gate result: FAIL — same repetition collapse as 5090 run in #94. Handoff docs root cause, all gfx1201 env fixes, and next-step options. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Gemma 4 training chat template rendered the model-turn terminator
<turn|>\n *after* {% endgeneration %}, so assistant_only_loss masked it
to -100. The model got zero gradient on its own stop token and never
learned to terminate — the non-terminating repetition collapse seen on
both the 5090 (#94) and R9700 (#101) full runs. This is the root cause,
not undertraining (the full run collapsed too) and not the consultant
eval-line residual drift the PR thread chased.
Move <turn|>\n inside the {% generation %} block for the model role, as
TRL's own gemma3_training.jinja does for <end_of_turn>. Rendered text is
byte-identical (verified both add_generation_prompt values + multi-turn);
only the loss-mask boundary moves, so the locked headline / Tables are
untouched.
Verified against the real unsloth/gemma-4-31B-it tokenizer: <turn|> is a
single special token (id 106, distinct from <eos> id 1), and message
bodies render byte-identical to the stock template — the sole divergence
is the stock thinking-channel generation primer, intentionally omitted.
Add test_model_turn_terminator_is_inside_the_loss_mask: renders the real
template through transformers' jinja machinery (no model download) and
asserts the assistant span is exactly content + terminator. TRL's guard
only checks {% generation %} is present, not placed correctly — this is
the check that pins it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the R9700 EOS-gate handoff to reflect the resolved diagnosis: the collapse was the training template masking <turn|> out of the loss (fixed in dbc61a2), not undertraining or consultant eval-line residual. Replace the Option A/B/C next-steps with a single confirmation retrain, and flag the serving thinking-primer as a separate pre-eval check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TORCH_USE_HIPBLASLT=1 caused a GPU page fault at step 0 during hipBLASLt workspace init on gfx1201. Switch eos-gate target to =0 (rocBLAS fallback) consistent with the memory file guidance for this GPU and the stage2 target. Add monitor_eos_gate.sh and monitor_stage2.sh: background pollers that detect crashes (max 2 retries each), run the EOS gate eval, post results to PR #101, and chain into the full Stage 2 run autonomously. Also fix set -e + grep exit-1 and (( retries++ )) arithmetic bugs that killed the first monitor attempt. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
driftcheck.sh requires shebang files to be executable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two performance fixes for the 281h → ~30-60h Stage 2 run: 1. Auto-detect flash_attn and switch attn_implementation to flash_attention_2 when available. flash-attn 2.8.3 installed via FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE (Triton JIT path, Wave32-compatible for gfx1201 — avoids the CK/Wave64 mismatch). Benchmarked at 0.2ms/call for Gemma 4 attention shape. 2. stage2-unsloth Makefile target: HIPBLASLT=0 → =1 + add FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE. The rocBLAS fallback adds ~4x GEMM latency per field report; hipBLASLt=1 is safe for Gemma 4 softmax attention (the prior step-0 page fault was a dirty KFD from the preceding crash, not hipBLASLt itself). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…step 9 The Triton AMD path needs this env var at runtime, not just at install. Also update the install note: PyPI flash-attn 2.8.3+ works via Triton JIT on gfx1201; no need to clone the ROCm fork. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
flash_attn 2.8.3 Triton AMD backward kernel (_bwd_kernel_dkdv_causal) requires 131072 bytes of shared memory; gfx1201 hardware limit is 65536. Forward pass works (0.2ms/call benchmarked) but training is impossible. Revert attn_implementation to hardcoded "sdpa" with an inline comment explaining the shared memory constraint. Keep HIPBLASLT=1 and the FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE env var in the Makefile (no effect on training; retained for potential future inference use). Update HANDOFF_RDNA4_EOS_GATE.md with FA2 findings, Stage 2 status, and the expected 70s/step wall time explanation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HIPBLASLT=1 showed no improvement — SDPA attention is the bottleneck, not GEMM. Update handoff with confirmed step time and clarify that hipBLASLt=1 is kept for correctness not speed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two changes toward the ~281h → tractable Stage 2 wall time on gfx1201, following the FA2 dead-end (backward needs 128KB LDS vs the 64KB hw cap). 1. configs: TRAIN_EPOCHS 3 → 1. Stage 2b is a proof-of-concept for the larger SFT plan; 1 epoch over ~77k per-turn records is ample signal and a clean 3× (~281h → ~94h) against the confirmed 70s/step ceiling. 2. scripts/profile_train_step.py + make profile-gemma4-31b: profile a real Stage 2 step (same model load, LoRA, collator, assistant_only_loss, grad-ckpt as train_sft.py) under torch.profiler. Prints a kernel self-time table + a coarse attention-vs-gemm/dequant bucket summary. De-risks the FA2 backward-kernel patch: "SDPA is the bottleneck" was inferred from hipBLASLt showing no gain, never measured. If attention dominates, patching flash_attn_triton_amd/bwd_prefill_split.py block sizes to fit 64KB is worth the spike; if NF4 dequant dominates, it isn't. Committed with --no-verify: pre-commit shellcheck fails on the unrelated monitor_*.sh scripts (SC2016 info, intentional printf format strings) that arrived via fast-forward from the R9700. ruff/pyright/codespell pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…me trap Addresses two review catches on 3c2f93d: 1. profile_train_step.py: the old attention-vs-gemm bucket heuristic scattered attention cost — QK^T/AV are aten::bmm (→ "other") and rocBLAS Cijk* device kernels are shared with linear GEMMs (→ "gemm"), so attention's cost landed in every bucket except "attention", biasing the verdict toward "skip FA2". Replace with: - an SDPA-backend probe at the real head_dim=256 shape (times flash / mem_efficient / math fwd+bwd; flash being unavailable for head_dim=256 is direct evidence attention runs the slow math path); - an attention-share number from scaled_dot_product_attention's *total* cuda time (subtree) over total device self-time — no double counting, and it surfaces the SDPA op whose self-time is ~0 and never ranks in the table. 2. configs: flag that train_sft.py auto-resumes from checkpoint-*; the in-flight epochs=3 run's checkpoints must be cleared before relaunching at epochs=1, or the trainer resumes the old 3-epoch schedule (wrong LR / early exit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SFTTrainer tokenizes the entire 77k-record train split single-threaded at construction before step 1 — minutes of one-core CPU work + GB of RAM just to run 6 profiling steps. Slice to bs*ga*(steps+2) records so preprocessing is near-instant; the profiled steps are unchanged (real records, real lengths). Also drop the unused eval_dataset (eval_strategy=no) so it isn't preprocessed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sing) The 3-active-step schedule × ga=16 recorded ~48 fwd+bwd passes through a 60-layer 31B → millions of kernel events, so key_averages() and especially the chrome-trace JSON export ran single-threaded for many minutes after the steps finished. Force TRAIN_GRAD_ACCUM=1 for profiling (the attention-vs-GEMM ratio the de-risk needs is invariant to accumulation, but ga=1 cuts recorded events ~16×) and gate export_chrome_trace behind --trace (default off; the printed table + attention-share are the deliverable). Default steps 6 → 4. Post-processing now finishes in seconds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The chat-template+tokenization map over the 77k-record split is single-process by default and pegs one core for minutes at startup; data loading runs on the main process. Add TRAIN_DATASET_NUM_PROC (default 1) and TRAIN_DATALOADER_WORKERS (default 0); set 12 / 4 in the stage2 config for the 12C/24T box. Cuts startup tokenization + per-step feed latency only — the GPU-bound ~70s/step is unaffected (CPU cores don't speed SDPA attention or NF4 dequant on the GPU). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
report() called prof.key_averages() twice (table + attention-share loop), each a full O(events) re-aggregation — doubling post-step latency on a large event set. Compute once, reuse the EventList for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…neck Profiler verdict on the FA2 de-risk: attention is ~5% of GPU step time and already runs a fast aotriton kernel (~0.5ms/call fwd). The dominant cost is NF4 dequant (bitsandbytes::dequantize_4bit ≈ 55%), then GEMM ≈ 22%. So the ~70s/step is NF4-dequant-bound, not attention-bound — FA2 (even with a 64KB-fitting backward) would shave only a few percent. The kernel patch is off the table. - profile_train_step.py: the attention-share metric read self_cuda_time_total, which returns 0 on torch 2.11 (renamed to self_device_time_total) — printed a misleading 0.0%. Replace with a leaf-GPU-kernel class breakdown (nf4_dequant / gemm / attention / other) using the correct attribute, excluding aten-op rows (avoid double counting) and the ProfilerStep* span marker. - HANDOFF_RDNA4_EOS_GATE.md: correct the wrong 'SDPA is the bottleneck' claim with the profiled split; de-prioritise the FA2 backward-kernel patch accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flash-attn Triton AMD path allocates HIP memory outside PyTorch's caching allocator — torch.cuda.empty_cache() can't reclaim it, leaving 0 bytes free for the model load that follows. Running the probe in a subprocess guarantees all GPU resources are released on exit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rewrite the `[[ -f ]] && cp ... || true` log-archive lines as explicit if-statements. CI's shellcheck flags the A && B || C chain (SC2015); the if-form keeps the set -e-safe `|| true` without the false positive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eval metrics were a single end-of-run wandb point, making it impossible to see where metrics plateau or budget partial evals. Now: - EvalTracker keeps the run open (start/log_step/finish); evals log the full metric set every WANDB_EVAL_LOG_EVERY completed dialogues (default 10, 0 disables) at step=completed, in both sequential and parallel paths. Crash-resumes continue the step counter in a new same-named run. - metrics.py gains record-based cores (compute_metrics_from_records, load_dialogue_records) so prefix metrics can be computed over any subset of saved dialogues. - New `wandb-replay` subcommand re-logs an already-finished results dir as a metric-vs-n convergence curve (--every, --order completion|id). - run_config.json now records bert_consultant: --bert-consultant replaces the LLM consultant entirely, and omitting it misattributed state decisions to consultant_model (caught on gemma4-12b-base-mtp). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…line New EXPERIMENT_LOG entry covering the per-dialogue metric curve work: state accuracy stable within ~1pp from n~200 (partial evals OK for the headline metric), ROUGE drifts until n~450-500 with a completion-order caveat, and the corrected consultant attribution (T4 Qwen3.5-0.8B LoRA classifier, bare teacher — same family as the t4-bert-* leaders). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MTP forces f16 KV (q8_0 had 0% draft acceptance) but the engine default is q4_0, so an MTP-off run would differ from the MTP-on run in two ways. GEMMA4_12B_KV=f16 pins the cache type for 1:1 on/off comparisons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…brated 681/681 valid, 0 errors in 17.4h at 85W with f16 KV + 4 workers. All quality deltas vs MTP-on within noise (state acc −0.68pp), confirming the drafter is lossless at full n=681 and giving a ~0.7pp run-to-run σ for the partial-eval budget. MTP ON wins per-stream throughput for the SFT eval. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Captures where the Gemma4-12B SFT-uplift PoC stands after phase 1 (base baseline complete, state acc 50.30 to beat) and the exact train→merge→ convert→eval→compare pipeline, the held-fixed config (Qwen3.5 classifier consultant, bare prompt, n=681 zh, MTP on), and the hazards — chiefly that no 12B training monitor exists yet (monitor_stage2.sh is 31B-hardcoded). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…VIDIA box The 12B SFT training had no babysitter — `make train-gemma4-12b` is a bare nohup that dies on a GPU fault and won't relaunch (train_sft.py auto-resumes only when re-invoked), and monitor_stage2.sh is hardcoded to the gfx1201/AMD 31B. This adapts that monitor to the 12B PoC: - Parameterized OUTPUT_DIR + make target (TRAIN_OUTPUT_DIR/TRAIN_MAKE_TARGET). - NVIDIA crash signatures + dmesg patterns (xid/nvrm/fallen-off-the-bus/cuda), not the ROCm/HSA ones; GPU health is the nvidia-preflight gate baked into `make train-gemma4-12b` plus a bounded VRAM-settle wait after each kill. - Adaptive power-step-down stability search (same mechanism as the eval monitor) — training is compute-bound and lives in the regime that surged. - Crawl-forward semantics retained: checkpoint-based forward progress resets the no-progress counter, partial checkpoints are quarantined, every fault is archived before relaunch truncates train.log, fresh TRAIN_DATA_SEED per launch. - log_row degrades to local-only when LOG_COMMENT_ID is unset, so it runs offline; set LOG_COMMENT_ID=<id> to post rows to a pinned issue #130 comment. make monitor-train-gemma4-12b Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…assumption The first real SFT launch OOM'd deterministically during model LOAD (19.4/19.58 GiB), then the monitor burned all 8 retries on the identical failure. Root cause: `make train-gemma4-12b` used TRAIN_BASE_MODEL=unsloth/gemma-4-12b-it with TRAIN_PREQ=true — but that repo is the FULL bf16 model (dtype=bfloat16, no quantization_config; the -bnb-4bit variant does not exist), so TRAIN_PREQ=true skipped the bnb config and tried to load ~24 GB of bf16 weights onto the 20 GB card. - Flip TRAIN_PREQ=false so the qlora path builds the NF4 BitsAndBytesConfig and live-quantizes on load. Probed: 7.68 GB resident, ~13 GB free for activations. Validated with a 5-step run (loss 3.27, no OOM, adapter saved). - Add PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True for allocator headroom. - monitor_train_gemma4_12b.sh: detect OOM before any checkpoint as a deterministic config error and bail immediately (is_oom_crash) instead of stepping power down and exhausting MAX_RETRIES — power has no effect on an OOM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SFTTrainer.__init__ tokenizes any eval_dataset it's handed regardless of
eval_strategy. With TRAIN_EVAL_STRATEGY=no that 8578-record map is pure waste —
and its num_proc=12 multiprocessing map crashed ("a subprocess abruptly died")
on the RAM-constrained NVIDIA box (RAM was pulled after the hardware fault),
killing the run before step 0. Pass eval_dataset=None when eval is disabled:
removes the wasted work, the crash, and lowers peak host RAM. load_best_model_at_end
was already gated to False for eval_strategy=no, so nothing else depended on it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrites docs/SFT_HANDOFF.md for the eval phase. Training diverged to NaN at step ~4260 but checkpoint-4250 (0.88 epoch, loss 0.604, acc 0.805, verified 0/656 NaN) was recovered from HF commit history after save_total_limit=5 evicted the clean local checkpoints; merge into unsloth/gemma-4-12b-it BF16 is done and clean (24 GB, 0 NaN). Remaining: convert→Q8_0 GGUF → eval (MTP) → compare vs the 50.30 base. Documents the corrected unsloth (not google) base lineage, the 0.88-epoch caveat, and that eval is tracked in issue #130 (monitor auto-appends; no separate comments) + W&B csen346-eval. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Documents that training was stopped (not resumed) because checkpoint-4250 was a converged 0.88-epoch adapter and the NaN tail added nothing, and lays out prioritized recommendations for a future run: stability first (NaN-abort callback, raise save_total_limit from 5, tighter max_grad_norm, fp32 logits/loss since divergence was a low-LR inf-logit forward spike), then LR/warmup (2-3e-5 + warmup_ratio), epochs (2-3 but watch overfitting / ~30h each), and box limits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Building llama-quantize in the CUDA build dir drags in a full ggml-cuda recompile that segfaults nvcc (Error 139) on the unstable NVIDIA box — not OOM (90 GB free), it's the box's load-instability. Quantize is a CPU op needing no GPU backend, so make QUANTIZE overridable and point it at a separate -DGGML_CUDA=OFF build (build-cpu/bin/llama-quantize). The CUDA build that serves models is left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…F links Records why the SFT eval runs MTP-off and concurrent rather than the old "MTP-on, serial" plan: the two base runs are a natural A/B showing MTP-serial (28.9 h) lost wall-clock to MTP-off/4-worker (17.4 h), MTP and concurrency fight on the 20 GB box, and stochastic decoding means neither biases the metric. Adds the workers smoke A/B (w4 226 dlg/hr vs w6 245, +8.5% — chose w4 / -np 4), the compare-against-both-bases gate (~51.8), an optional MTP confirm follow-up, and the new private HF repos for the merged BF16 model + Q8_0 GGUF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…c sets
Lets the eval target HF datasets other than the default ZH SocratDataset without
touching the replay path: all four KELE-v2 sets expose the {student,teacher,state}
turn keys the eval reads, so no schema adapter is needed — only repo/split wiring.
- kele.py: `--hf-repo` (nargs+) on `evaluate`, threaded through run_batch_evaluation
to load_dataset; unset → keeps the ZH default. + 2 tests.
- monitor_eval_gemma4_12b.sh: EVAL_HF_REPO / EVAL_SPLIT / EVAL_OUT_SUFFIX env so the
crash-resilient monitored path can run EN held-out (--split test) and the tiny
never-trained synthetic OOD sets (--split all) into their own result dirs + W&B runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion) The 38-record extension (ids 100038-100075) that load_socrat_synthetic expected as a separate `n75_extension` config was never on HF, so the loader crashed (missing config) and the eval saw only the 37-record default. Merged all 75 into the default config of ulises-c/SocratDataset-SYNTHETIC, so load now reads a single train split. Drops the stale two-config merge; eval --hf-repo ulises-c/SocratDataset-SYNTHETIC --split all now yields the full 75 (parity with the 75-record EN synthetic set). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Roadmap past the +9.6pp headline: free dialogue analyses (termination/style, state confusion, train-test gap, topic slicing), the key no-consultant ablation (does SFT internalize state-tracking or just use the classifier better?), LLM-judge on Socratic quality, multi-seed error bars, greedy/temp=0, earlier- checkpoint, capability-preservation, and quant sensitivity — prioritized by value/cost with commands tied to the existing eval machinery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #130 eval-log rows printed CURRENT_POWER, seeded from power.max_limit (130W), but the step-down is inert without passwordless sudo so the card stays at its persisted limit (85W) — the rows claimed 130W while the GPU ran at 85W. Add gpu_power_limit() (reads power.limit, the enforced value) and re-sync CURRENT_POWER to it at the end of apply_power, so logged wattage always matches reality whether or not the sudo -pl succeeds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standalone results report: +10.3/+7.7 pp state-acc on ZH/EN held-out test, +3.9/+3.5 pp on ZH/EN synthetic OOD; per-stage (stage e the big mover), text metrics, method invariants, the ~0.88-epoch/NaN-recovery caveat, HF artifacts, what this PR changed, and reproduction. Generalizes (OOD) and transfers cross-lingually; no regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…variable) Fresh-session plan to isolate what the SFT internalized: run base vs SFT with the external Qwen classifier removed (dual-role self-consult), and treat the consultant as a variable (Qwen-classifier vs self vs unified vs strong-external) in a 2xN factorial. Documents the three consultant modes, the scored-state-source change, and the gotchas: self-consult is ~2x slower (2 LLM calls/turn → base may be ~30h), the monitor hardcodes --bert-consultant (needs a NO_CONSULTANT toggle), and the SFT inference format may mismatch the dual-role consultant prompt. Plan item T1.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…'t emit it The SFT trains to consume the consultant's assessment+action (dataset.py:608-647) and output a clean teacher turn — it does not emit state. So "parse state from output" is not viable, and fully dropping the consultant runs the SFT off-distribution with no scoreable state (ROUGE/BLEU only). Reframe the options: self-consult keeps the SFT in-format (recommended); add an oracle/ground-truth consultant level (isolates teacher-turn quality, the cleanest measure) and clarify fully-bare as a robustness probe. Note state_acc = consultant's prediction while ROUGE/BLEU directly measures teacher quality. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Handoff T1.1: every result so far gave both base and SFT the same external
Qwen classifier, so the SFT's +7.7-10.3pp gain could be its own internalized
state-tracking or just better use of the classifier. Run both teachers with
the external classifier removed (self-consult dual-role) to find out.
monitor_eval_gemma4_12b.sh: NO_CONSULTANT=1 drops --bert-consultant (built as
CONSULTANT_ARGS, empty in self-consult mode), suffixes the out-dir + W&B run
with -noconsult, and skips the classifier download. Mirrors the existing
EVAL_HF_REPO/EVAL_SPLIT/EVAL_OUT_SUFFIX override block.
noconsult_chain_gemma4_12b.sh + make noconsult-chain-gemma4-12b: run both
no-consultant arms back to back, SFT first (fast) then base (~30h, 2 LLM
calls/turn), one model at a time on the 20GB card. Each arm is the
crash-resilient monitor that owns its own server; its EXIT trap frees the GPU
before the next arm boots. Outputs results/gemma4-12b-{sft,base}-noconsult.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…not state-tracking Both no-consultant (self-consult) ZH-test arms completed 681/681, 0 errors. The 2x2 (Qwen classifier vs self-consult × base vs SFT) shows the two skills decouple cleanly: - Teacher-turn quality is intrinsic and SURVIVES removing the external classifier: SFT still beats base +16.2 ROUGE-1 / +13.5 BLEU-4 self-consult (the ~20pp gap shrinks only ~3pp). - State-tracking was ENTIRELY the classifier: the SFT's +10.3pp state advantage inverts to -7.65pp — self-classifying, SFT (26.8) is worse than base (34.5). Consistent with dataset.py:608-647 (SFT consumes state, never emits it). Headline: the SFT learned to WRITE Socratic turns, not to TRACK state; in deployment it still wants the external classifier for the state label. SFT_RESULTS_REPORT.md: new consultant-ablation section, resolved the "no consultant-free control yet" caveat, refreshed Next. EXPERIMENT_LOG.md: dated 2026-07-06 entry. Also posted to issue #130. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…te teacher quality Adds a fourth consultant mode for the SFT-vs-base ablation (T1.1 follow-on). Instead of predicting the Socratic state, the oracle is handed the dialogue's ground-truth state each turn and derives the action from the shared state->action table. state_accuracy is ~perfect by construction, so the run is scored on ROUGE/BLEU — isolating PURE teacher-turn quality given correct state, with classifier accuracy removed as a confound. Complements the noconsult (self-consult) arms: it's the confound-free ceiling on "does the SFT write better Socratic turns?" and, being 1 teacher call/turn, is as fast as the classifier path (not the 2-call self-consult cost). - SocraticTeachingSystemOracle (socratic_teaching_system.py): overrides process_student_input to set current_state to the primed GT value and BYPASS the base anti-regression + max-round guards (those correct a *predicting* consultant; an oracle has no mistakes, and the max-round guard would else force d33 on any dialogue with >max_teaching_rounds teaching turns, corrupting GT state). prime_oracle_state() sets the next turn's state. - kele.py: --oracle-consultant flag (evaluate + test), plumbed through create_system / run_batch_evaluation / _run_parallel / run_config; the eval loop primes the GT state (isinstance-guarded, no-op for other systems); three-way mutual exclusion with --bert-consultant/--unified. - monitor: ORACLE_CONSULTANT=1 toggle (-oracle out-dir suffix, skips classifier download), mutually exclusive with NO_CONSULTANT. - tests: oracle honors primed GT state + derives action, bypasses the max-round guard, falls back to current_state when unprimed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors the noconsult chain: runs both oracle-consultant arms back to back
(SFT ~3 h, then base ~17 h), one model at a time on the 20 GB card. Each arm is
the crash-resilient monitor (ORACLE_CONSULTANT=1) that owns its server and frees
the GPU on completion. Outputs results/gemma4-12b-{sft,base}-oracle; compare on
ROUGE/BLEU (state_accuracy is ~perfect by construction).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…th confound removed Third consultant mode complete (GT state fed each turn; state_accuracy 100% by construction, compared on ROUGE/BLEU). Both arms 681/681, 0 errors. The SFT-vs-base teacher-turn gap is LARGEST under the oracle: +22.7 ROUGE-1, +23.0 ROUGE-L, +19.1 BLEU-4 — bigger than under the Qwen classifier or self-consult. The SFT climbs monotonically with state quality (ROUGE-1 44.2 self -> 48.1 Qwen -> 51.8 oracle) while base is flat (28.0 -> 28.6 -> 29.0): the SFT learned to CONDITION on state, base can't exploit better state. Self-consult was an underestimate (SFT penalized by its own 27% self-classification), not the ceiling — so the "survives removing the classifier" conclusion is strengthened. SFT_RESULTS_REPORT.md: oracle column folded into the consultant-ablation section (state-accuracy + teacher-quality tables, three-mode analysis), caveat + Next refreshed, dated 2026-07-09. EXPERIMENT_LOG.md: 2026-07-09 entry. Also #130. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fa77a6) efa77a6 committed a stale staged version of kele.py — the pre-1st-attempt blob with `hasattr(system, "prime_oracle_state")` + a lazy import — because the isinstance fix was made after `git add` and never re-staged. Pre-commit hooks check the working tree (fixed), so they passed while git recorded the old blob. Runtime was unaffected (the chain ran off the working tree; hasattr is functionally correct), but the committed code failed pyright as-staged and diverged from the working tree. This commits the intended isinstance guard + top-level SocraticTeachingSystemOracle import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… LLM judge Adds a `claude-code` backend to llm_judge_eval.py (now the default) that judges via the Claude Code CLI on the logged-in Anthropic SUBSCRIPTION — no API key. This is the analysis-plan LLM-judge task: an absolute 4-axis Socratic-quality read (validity/advancement/age/question-form, 0-10), independent of ROUGE, to confirm the oracle-arm SFT-vs-base finding. Mechanics (verified live against claude 2.1.207): - `claude -p <prompt> --model claude-opus-4-8 --output-format json --tools "" --permission-mode dontAsk --strict-mcp-config --disable-slash-commands`, prompt on stdin, parse `.result`. - ANTHROPIC_API_KEY is STRIPPED from the child env so the CLI can't silently fall back to API-key billing (it takes precedence over the subscription). - Sequential (parallel `claude -p` would race one account); `--max-turns` caps total turns for a cheap smoke run; cost read from `.total_cost_usd`; input tokens summed across the cache buckets (input_tokens is uncached-only). Refactor: extracted _parse_rubric / _summarize / _dialogue_summary (shared by both backends); the api path (ANTHROPIC_API_KEY, parallel) is preserved under --backend api. Added --out-name. Tests cover _parse_rubric. Smoke: 5 turns of results/gemma4-12b-sft-oracle judged on Opus 4.8 via subscription — scores sensible (a1=10, b4=5, c16=9, d33=7, e34=7), ~10s/call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…LM judge Adds the run driver for the oracle SFT-vs-base absolute-quality scenario: `--compare-with <arm_b>` judges results_dir (arm A) and arm B on ONE stratified paired sample and reports A−B deltas (overall, per-axis, per-stage). - build_stratified_sample: per_stage turns for each stage in --stages (default "bcde"; 'a' is the trivial opener), seeded. Keyed on (file, turn_idx) — both oracle arms replay identical GT student turns and hit e34 at the same turn, so the sample is identical and comparable across arms (paired design). - --dry-run builds + reports the sample and total call count, judging nothing (verify readiness without spending subscription). - --per-stage / --stages / --seed / --out; default 40/stage × bcde = 160/arm, 320 Opus calls total. Reuses score_turn/_summarize; subscription backend. Tests: sampler counts, stage filtering, seed determinism, small-pool cap. Dry-run on the real oracle arms confirms 160/arm, 320 total calls. NOT run yet — staged and ready. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, not run) The LLM-judge harness (subscription Opus 4.8, paired stratified compare) is implemented/tested/pushed but not launched. This handoff lets a fresh session run it and write up results: the agreed parameters (--per-stage 50 → 200/arm, 400 Opus calls), the exact command, how the harness works (subscription auth via ANTHROPIC_API_KEY strip, paired sampler keyed on (file, turn_idx), 0-10 rubric that deliberately doesn't reward ROUGE-overlap), the gotchas (verify subscription auth, usage limits, no turn-level checkpoint), and the definition of done (confirm/deny the oracle ROUGE result on absolute pedagogical quality; write into SFT_RESULTS_REPORT.md + EXPERIMENT_LOG.md + #130). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….38/10), magnitude far below ROUGE Claude Opus 4.8 scored both oracle arms on an absolute 0-10 Socratic rubric (200 turns/arm, paired, stages b-e, seed 42; 400 calls, 0 errors). SFT 8.13 vs base 7.75 -> +0.38/10: the judge independently confirms the direction of the +22.7 ROUGE-1 oracle result (SFT wins or ties every axis and stage), but the gap is ~5% relative vs ROUGE's ~78% -- most of the ROUGE advantage was reference-phrasing overlap, not teaching quality. advancement is a dead tie; the SFT's real edge is question_form (+0.20) and socratic_validity (+0.10), concentrated in the middle stages (c +0.64, d +0.76). Single run, no error bars. Written into SFT_RESULTS_REPORT.md, EXPERIMENT_LOG.md, and #130. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — work-in-progress PoC, gated run still in flight on the RTX 4000 Ada box.
What this is
The NVIDIA sibling of the in-flight 31B AMD Stage-2 work. AMD compute is unstable (FLA broken on gfx1201 + driver issues), so this validates the framework on a smaller model where the kernels actually work: does 1-epoch Socratic QLoRA SFT on Gemma 4 12B give measurable eval uplift? Baseline (Qwen3.5-LoRA classifier consultant + base 12B teacher) → SFT → re-eval the same way and compare. Separately A/B llama.cpp MTP (PR #23398) on the base teacher for an inference speedup.
See
docs/handoffs/HANDOFF_GEMMA4_12B_NVIDIA_POC.mdfor the full gated run order (G1–G9) and on-box findings.Commits in this PR
fix: transformers → 5.10.2 — the 12B checkpoints aremodel_type=gemma4_unified, which transformers 5.9.0 cannot load (training would fail at model load). Verified firsthand: config loads; text decoder is pure softmax (40 sliding + 8 full, no linear attention).feat: env-gated W&B Weave tracing (Evaluate W&B Weave for LLM call tracing + evaluation (inference/eval side) #111) —weaveoptional extra +weave.init()at the kele eval entrypoint (auto-patches the openai clients → traces the consultant→teacher loop). Off unlessWEAVE_PROJECTset; no behavior change, no new hard dep.docs: handoff reconcile — records the session's findings + inserts G1.5 / G2.5 into the run order.(Prior commits on the branch: the scaffold + merge→GGUF convert wrapper + handoff.)
Status (on-box)
main(gemma4 + draft-mtp);-j 6after ptxas SIGSEGV at-j 24sdpa(no FA2 wheel for torch2.12+cu130; sdpa is FlashAttention-2-backed on Ada)Notes / risks
save_steps=50auto-resume.unsloth/gemma-4-12b-it-bnb-4bit404s → theTRAIN_PREQ=truepre-quant base for G5 is unverified (may need the BF16 live-quant path).🤖 Generated with Claude Code