d

d/ — Full Reconciliation Pipeline

d.ipynb is a reorganization of c/ — it applies the same core reconciliation logic in a cleaner, production-ready order, fixes one limitation of c (family regroup is done upfront instead of mid-pipeline), and adds a GPT-5.1 vision step for cases that rule-based logic cannot resolve. All transformations operate in-memory on gpt_page_dict and the final result is exported as d.csv.

As in c, A refers to the CV predictions (a_2_cv_result.csv) and B refers to the GPT predictions (b_trans_gpt.csv).

---

Pipeline Steps

Step 1 — Load CV results

Reads a_2_cv_result.csv and builds per-page summaries: cv_staff_cnt, cv_system_list, cv_group_list.

---

Step 2 — Load GPT results and re-derive staffgroup by instrument family

Reads b_trans_gpt.csv, forward-fills the page column, and maps each staff's instrument to a family (Woodwind, Brass, Percussion, Strings). Staffgroups are immediately re-labelled by consecutive family runs, replacing raw GPT group IDs.

This is the family-regroup logic from c_4, but done at the start rather than mid-pipeline, so all subsequent steps work with cleaner groupings.

---

Step 3 — Auto-remove excess staves where B > A (same-length run-lists)

For pages where A and B have the same number of groups but B has too many staves, applies the same four rule-based removal operations as c_1:

• Adjacent duplicate rows
• Adjacent subset rows
• All-NaN rows
• NaN wildcard subset rows

A block-level guard prevents over-removal.

---

Step 4 — Truncate pages where B's run-list is a repeated prefix of A

For pages where GPT hallucinated a repeated layout (e.g., GPT group list [4, 5, 1, 5, 4, 5, 1, 5] vs CV [4, 5, 1, 5]), truncates the GPT rows to the first len(A_runs) groups.

This corresponds to c_6.

---

Step 5 — Supplement missing staves using bracket data where B < A by 1

For pages where B is short by exactly one staff in a group (same-length run-lists), loads bracket detection data from the bracket JSON files. If a bracket spans the expected gap, inserts a synthetic row copied from the bracketed neighbor instrument.

This is a more precise version of c_2: rather than blindly padding, it only supplements when bracket evidence confirms a real staff is missing.

---

Step 6 — GPT-5.1 vision fix for remaining discrepancies

For pages that still disagree after all rule-based steps (staff count differs AND same-length run-lists), calls the GPT-5.1 vision API via gpt_fix_group(). Each call sends the current HTML table plus the score image, and instructs GPT to merge or delete rows until specific group sizes match CV.

This step replaces c_5 (force-merge) and handles the cases that rule-based logic cannot confidently resolve.

---

Step 7 — Export d.csv

Normalizes part values (e.g., 2.0 → 2) and concatenates all pages from gpt_page_dict into a single CSV (192 pages, ~3262 rows).

---

d_eva.py

Evaluates d.csv (or any pipeline output) against a ground truth CSV. Reports across three scenarios (all pages, correct-staff-count pages, incorrect-staff-count pages):

• System boundary P/R/F1
• Staff group boundary P/R/F1
• Instrument / Part / Tone accuracy with tone alias normalization (e.g., "B flat" = "bes")

Saves a mismatch report to d_eva_mismatch.csv.