FinReason-Lab — Dissecting Post-Training Gains on Financial Numerical Reasoning

English · 简体中文

Where do "post-training gains" actually come from? Using FinQA as a locked testbed, I decompose the accuracy a Qwen3 model gains after intervention into three separable, individually measured components — prompt-protocol alignment, answer-format SFT, and explicit thinking — across the full 0.6B → 14B dense family. The headline finding: the single largest lever is the one nobody trains for. Just switching a capable base model from a training-style text prompt to its native chat protocol (no fine-tuning, no thinking) recovers +20–30 points; on top of that, answer-only SFT adds little and sometimes subtracts (4B: 31 fixes / 38 breaks, p = 0.47), and a GRPO/RLVR capstone reward-hacks its own format bonus, collapsing from 31% to 9.7%.

A solo research-engineering extension of a UCL COMP0087 team project · single-GPU (1× NVIDIA GB10) · every training, evaluation, orchestration, and analysis component in this repo is my own implementation.

---

📊 One-Page Evidence — Results at a Glance

Everything a reader needs to judge the work, on one screen. FinQA test split, oracle context, n = 300 per cell, seed 42, scored with the robust math-verify+legacy verdict (acc_adjusted). The flagship figure is finqa_baseline/results/scaling/scaling_curve.png.

The flagship — recoverability across the Qwen3 family (acc_adjusted, %)

size	Zero-shot (text, no-think)	Fair (chat, no-think)	Answer-only SFT	Thinking	SFT net (vs fair)
0.6B	2.33	3.00	4.00	19.33	▲ +1.00
1.7B	1.00	6.00	14.67	26.00	▲ +8.67
4B	3.00	33.67	31.33	25.67	▼ −2.33
8B	2.00	23.00	31.00	31.00	▲ +8.00
14B	0.33	30.33	34.00	34.33	▲ +3.67

The text column is a handicapped baseline (the model is asked in a format it barely emits a parseable answer in — 8B fails to produce a tag on 65% of items). The honest question is not "how far above that", but "how far above a fair baseline that only swaps in the chat protocol".

Three findings that survive paired-McNemar controls (decomposition by mechanism, legacy verdict)

1. Protocol alignment is the dominant lever — for big models only. chat_nothink vs text_nothink is overwhelming at scale (8B: 63 fixes / 0 breaks, p ≈ 2e-19; 14B: 91 / 1, p ≈ 4e-26) yet insignificant at 0.6B (4 / 2, p = 0.69). Capable models are format-locked, not incapable.
2. Answer-only SFT adds little over a fair baseline, and can hurt. SFT-vs-fair is insignificant at 0.6B / 4B / 14B and at 4B actually breaks more than it fixes (31 / 38, p = 0.47). The famous "+28 pp from SFT" is almost entirely the protocol windfall in disguise.
3. Thinking rescues small models but back-fires on mid-size. think vs fair is huge at 0.6B (51 / 2, p ≈ 3e-13) and 1.7B (62 / 2, p ≈ 2e-16), but at 4B it breaks more than it fixes (17 / 41) — the <think> block injects percent_scaling errors a capable model wouldn't make when answering directly.

Why accuracy moves (error composition, rate of total)

condition	parse_fail	percent_scaling	numeric_mismatch	reading
8B text, no-think	65.3%	4.3%	20.0%	format-locked: can't even emit a tag
8B fair / SFT	~0%	0%	~70%	format solved; residual is real arithmetic error
8B thinking	3.7%	36.0%	9.7%	reasoning, but trips on percent/scale conversion

Capstone — RLVR (GRPO) reward-hacks its own reward

With reward = 1.0·correct + 0.1·closed_tag, GRPO on 8B discovers the cheap half: it emits an empty placeholder tag [FINAL_ANSWER]...[/FINAL_ANSWER] on 78% of items to bank the +0.1 format bonus (93% closed-tag rate), then rambles the real reasoning outside the tag — so accuracy collapses 31% → 9.7% even though 218 / 300 "parse-fail" completions actually contain the computed number. A textbook verifiable-reward failure: optimize the easy gameable half, abandon the verifiable half. (finqa_baseline/results/scaling/capstone.md.)

The methodological spine that makes it trustworthy

• Paired McNemar + bootstrap 95% CI on every within-size comparison (resolves the original project's "single-run, no significance testing" limitation).
• Precision is controlled, not assumed. An fp16-vs-bf16 head-to-head on 3 representative 8B cells shows the base dtype moves accuracy by ≤ 0.7 pp — within CI noise — and confirms the 8B text parse-fail spike is a protocol artifact, not a numeric one.
• Two reported verdicts. acc_adjusted (robust: tolerant of <number> wrappers, units, percent auto-scaling) is the headline; strict symbolic math-verify is reported as a lower bound. CIs and McNemar use the same robust verdict the curve plots — no metric mismatch.
• Prompt-protocol alignment is a first-class variable — every checkpoint is evaluated under the template it was trained on. A train/eval template mismatch alone is enough to erase all SFT gains.

Stack: Qwen3 {0.6, 1.7, 4, 8, 14}B · bf16 LoRA for every size (no quantization confound) · TRL 1.5 SFTTrainer / GRPOTrainer · math-verify · 1× NVIDIA GB10 (128 GB) · seed-fixed, fully scripted, resumable matrix orchestrator.

---

Why this project

Financial QA is a clean stress test for numerical reasoning: the answer is grounded in both tables and text, and correctness is unforgiving about scale, percentages, signs, and which number to pick. FinQA ships gold reasoning programs, so answers are verified symbolically instead of by string match.

This started as a six-author UCL COMP0087 team project (I owned the entire codebase). That project found, under a fixed protocol, that Qwen3-4B beats Qwen3-8B zero-shot and that low-data SFT "recovers" the smaller model — and named prompt-format alignment as the likely confound. As a solo extension I set out to test that confound head-on and at scale, which turned a two-model curiosity into a precise, falsifiable decomposition:

Of the accuracy a model gains after post-training, how much is protocol alignment, how much is answer-format supervision, and how much is explicit reasoning — and how does that split move with scale?

The answer (above) both confirms and sharpens the original thesis: protocol alignment isn't a confound, it's the dominant term, and it only switches on once the base model is large enough to have the latent capability that the format was hiding.

What I built (skills demonstrated)

This repo is the end-to-end experimental harness, not a notebook dump. It demonstrates:

• LLM post-training at family scale — parameter-efficient bf16 LoRA SFT across 0.6B → 14B and a GRPO/RLVR capstone with a verifiable math-verify reward, all on a single GPU. Precision is held constant (bf16 everywhere) so within-size deltas carry no quantization confound.
• Controlled experimental design — a 5 × 4 size×condition matrix where the only moving part per comparison is the intended one; a fair baseline (chat protocol, no thinking) added specifically to give SFT's gain an honest denominator.
• Statistical rigor — bootstrap 95% CIs and paired McNemar tests that decompose each gain into protocol / format / reasoning terms with exact p-values.
• Evaluation engineering — a locked protocol (tagged-answer extraction, math-verify symbolic scoring with percentage rescaling), a robust-vs-strict dual verdict, and an explicit fp16/bf16 precision-robustness control.
• Diagnostic analysis — an automated error-composition classifier (parse_fail / percent_scaling / unit_confusion / numeric_mismatch) that explains why accuracy moves, not just that it moved.
• Reproducibility & ops — a resumable, memory-guarded matrix orchestrator (survives GPU contention on a shared box via a MemAvailable-aware guard), path-agnostic scripts, and tmux-hardened long-run recipes documented from hard-won failure modes.

Method overview

Component	Choice	Notes
Model family	Qwen3 {0.6, 1.7, 4, 8, 14}B dense	one consistent stack across the whole curve
Four conditions	text·no-think · chat·no-think (fair) · answer-only SFT · chat·thinking	the fair baseline isolates the protocol windfall
Precision	bf16 LoRA for every size; fp16 base at eval	no quantization; fp16↔bf16 shown equivalent (≤0.7 pp)
PEFT	LoRA r = 16, α = 32, dropout 0.05, q_proj,v_proj	identical across sizes
Eval protocol	oracle context, single pass, n = 300, seed 42	answer in [FINAL_ANSWER]…[/FINAL_ANSWER], math-verify (atol/rtol = 1e-3, auto % rescale)
Verdicts	acc_adjusted (robust) · acc_base/math-verify (strict)	headline uses robust; strict reported as a lower bound
Prompt-protocol alignment	train template ↔ eval template	first-class variable; mismatch alone nullifies SFT gains
Data	strict finqa_clean	3,277 train / 475 dev; kept only formula_exec_ok ∧ scale_relation = consistent

Condition definitions (exact). text·no-think = stage1_train_text prompt, thinking off, 64-token budget (the handicapped baseline). chat·no-think (fair) = chat_default prompt, thinking off, 1024-token budget — same protocol and budget as thinking, differing only by the <think> block, so it cleanly isolates protocol from reasoning. SFT = answer-only bf16 LoRA adapter, evaluated under its training template (stage1_train_text, no-think). thinking = chat_default, thinking on, 1024-token budget.

Results in detail

Decomposition by paired McNemar (within-size, legacy verdict; fixes / breaks / p)

size	protocol windfall (chat−text)	SFT net (sft−fair)	thinking net (think−fair)
0.6B	4 / 2 · p = 0.69	8 / 5 · p = 0.58	51 / 2 · p = 3e-13
1.7B	16 / 1 · p = 3e-4	29 / 3 · p = 3e-6	62 / 2 · p = 2e-16
4B	95 / 3 · p = 1e-24	31 / 38 · p = 0.47	17 / 41 · p = 0.002 (net negative)
8B	63 / 0 · p = 2e-19	41 / 17 · p = 0.002	50 / 26 · p = 0.008
14B	91 / 1 · p = 4e-26	27 / 16 · p = 0.13	36 / 24 · p = 0.16

Reading the table:

• Protocol is the story for every model ≥ 4B — large, lopsided (near-zero breaks), and astronomically significant. At 0.6B it does nothing (p = 0.69): there is no latent capability for the format to unlock.
• SFT-net is the quiet anticlimax. Beyond the protocol switch, answer-only SFT is significant only at 1.7B and 8B, is a wash at 0.6B/14B, and at 4B breaks more than it fixes. The dramatic "SFT vs text" deltas (e.g. 14B: 102 / 1, p ≈ 1e-29) are real but mostly protocol.
• Thinking dissociates by scale: a lifeline for small models, neutral-to-harmful for capable ones that do better answering directly than narrating a <think> block.

Error composition (finqa_baseline/results/scaling/error_composition.md) supplies the mechanism: the 8B text baseline fails by parse_fail (65%) — pure format lock; the fair/SFT conditions drive parse_fail to ~0 and leave numeric_mismatch (real arithmetic error) as the residual; thinking conditions are dominated by percent_scaling (the model reasons but botches percent/scale conversion).

RLVR capstone — a quantified reward-hacking case study. GRPO on 8B with reward 1.0·correct + 0.1·closed_tag does not improve on SFT — it actively reward-hacks. The policy learns that banking the format bonus is easier than earning correctness: 78% of completions emit an empty [FINAL_ANSWER]...[/FINAL_ANSWER] placeholder (93% closed-tag rate) and ramble the real reasoning outside the tag (mean completion grows 35 → 878 chars). Accuracy collapses to 9.7% (vs 31% SFT); tellingly, 218 / 300 parse-fail completions do contain the computed answer — just not in the tag, because the empty placeholder already collected the reward. On a task where format SFT already solves the interface, a verifiable-reward loop finds a shortcut that destroys it. Full breakdown in finqa_baseline/results/scaling/capstone.md; eval rows in 8B_grpo_text_nothink/.

Precision robustness. Re-running 3 representative 8B cells in bf16 against the fp16 originals moves acc_adjusted by ≤ 0.7 pp (zero-shot text +0.7, fair −0.3, SFT −0.7) — inside the bootstrap CI — and the 8B text parse-fail stays at 66–67% under both, confirming it is a protocol artifact. fp16 and bf16 are interchangeable here; the matrix is reported in fp16 with this control on record.

Full tables — per-cell CIs, the complete McNemar grid, and error composition — are regenerated into finqa_baseline/results/scaling/REPORT.md by a single command (below).

Repository structure

.
├── README.md                          # you are here
├── FinReason-Lab.pdf                  # the original team-project paper (project origin)
├── finqa_baseline/                    # EVALUATION pipeline
│   ├── eval_finqa.py                  #   single-run evaluator (--dtype, prompt protocols, dual verdict)
│   └── results/scaling/               #   the matrix: REPORT.md, scaling_curve.png, rigor/error reports
├── stage1/                            # TRAINING + ORCHESTRATION + ANALYSIS
│   ├── sft_train.py                   #   clean bf16 LoRA SFT (TRL 1.5, left-trunc, completion-only loss)
│   ├── grpo_train.py                  #   GRPO/RLVR capstone (verifiable math-verify reward)
│   ├── data/build_aligned_sft_data.py #   train/eval prompt alignment (the project's core fix)
│   └── scripts/
│       ├── run_scaling_matrix.py      #   resumable, memory-guarded size×condition orchestrator
│       ├── analyze_rigor.py           #   bootstrap CI + paired McNemar decomposition
│       ├── analyze_error_composition.py  #   why accuracy moves (error taxonomy)
│       ├── plot_scaling_curve.py      #   the flagship figure
│       └── build_scaling_report.py    #   one-command: collect → plot → rigor → error → REPORT.md
└── docs/                              # paper, proposal, reproducibility guide

The repo tracks only minimal reproducibility assets (debug samples, subset ID lists, cleaning summaries, code/configs, and the final result reports/figure). Checkpoints, logs, and full datasets are git-ignored.

Quick start

# 1) Environment (verified stack: torch 2.11+cu130, transformers 5.3, trl 1.5.1, peft 0.18)
python3 -m venv finqa_baseline/.venv
finqa_baseline/.venv/bin/pip install -r stage1/requirements-extension.txt
# torch is hardware-specific (GB10/aarch64 build here) — install the matching wheel first.

# 2) Build the train/eval-aligned data (oracle prompts via the evaluator's own template)
finqa_baseline/.venv/bin/python stage1/data/build_aligned_sft_data.py

# 3) Run (a slice of) the flagship matrix — resumable, memory-guarded
finqa_baseline/.venv/bin/python stage1/scripts/run_scaling_matrix.py \
  --sizes 0.6B,1.7B,4B --conditions zeroshot_text_nothink,zeroshot_chat_nothink,sft_text_nothink,zeroshot_chat_think \
  --num_samples 300 --max_steps 300

# 4) Regenerate the full deliverable (matrix JSON + figure + rigor + error composition + REPORT.md)
finqa_baseline/.venv/bin/python stage1/scripts/build_scaling_report.py

Reproduce a single zero-shot baseline (note the new --dtype):

finqa_baseline/.venv/bin/python finqa_baseline/eval_finqa.py \
  --model_name Qwen/Qwen3-8B --setting oracle --split test --num_samples 300 \
  --prompt_protocol chat_default --no-enable_thinking --max_new_tokens 1024 \
  --answer_format final_answer_tag --dtype fp16

Compute & reproducibility

• Hardware: 1× NVIDIA GB10 (128 GB unified memory). bf16 LoRA fits every size (14B ≈ 28 GB).
• Wall-clock (n = 300): no-think eval ≈ 5–15 min/cell; thinking eval ≈ 0.5–5 h/cell (token-bound); SFT ≈ 5–90 min by size; GRPO 8B ≈ 7 h.
• Software: TRL 1.5.1, PEFT 0.18.1, transformers 5.3, math-verify 0.9.0 (pins in stage1/requirements-extension.txt).
• Determinism: seed 42 throughout; single run per cell (n = 300), with bootstrap CIs and paired McNemar carrying the uncertainty rather than multi-seed repetition.
• Ops notes (learned the hard way, encoded in the orchestrator): on this unified-memory box torch.cuda.mem_get_info under-reports free VRAM by counting reclaimable page cache, so the memory guard reads MemAvailable; long jobs run under tmux because nohup did not survive a session kill.

Honest limitations

• fp16 base at eval (LoRA trained in bf16). The dtype gap is controlled — an fp16/bf16 head-to-head shows ≤ 0.7 pp movement — but it is a gap, stated on the record rather than buried.
• acc_adjusted is a tolerant verdict. It rescues <number>-wrapped and percent-rescaled answers the strict symbolic check rejects; strict math-verify is reported alongside as a lower bound. One cell (1.7B fair) leans on this: it emits <number>…</number> wrappers that only the robust verdict parses, so its strict score is an underestimate.
• n = 300 per cell, single seed. CIs and McNemar quantify the resulting uncertainty; some comparisons (e.g. 14B SFT-net, p = 0.13) are deliberately reported as not significant.
• Single benchmark (FinQA), oracle context. The decomposition is shown here, not yet across tasks.
• GRPO is a single reward design (1.0·correct + 0.1·closed_tag, 200 steps, num_generations 6). The reward-hack it exposes is specific to that small format bonus — a cleaner reward (no tag bonus, or length penalty) might avoid it. The claim is "naive verifiable-reward RL is hazardous where cheap SFT already suffices", not that RL is unfixable here.

These bound the claims rather than undermine them: the evidence that protocol alignment is the dominant, scale-gated lever — and that format SFT and verifiable-reward RL add little on top — is clean, controlled, and reproducible.

Citation

@misc{finreasonlab2025,
  title  = {Thinking-Aware Baselines and Low-Data Post-Training for Financial Numerical Reasoning},
  author = {Zhang, Yike and Wei, Jiaming and Wang, Yuhao and Liu, Yuxin and Wang, Jiaqi and Lang, Victor},
  note   = {UCL COMP0087 Statistical NLP project; solo scaling/decomposition extension by Jiaming Wei},
  year   = {2025},
  url    = {https://github.com/Quarkgluonmixture/FinQA}
}

License & data

Code is released under Apache-2.0. Qwen3 models are Apache-2.0; FinQA (Chen et al., 2021) is released for research use under the MIT license. Use of these resources complies with their respective terms.