From c3e73981e47caca6676dec4eaa7495fb8507f316 Mon Sep 17 00:00:00 2001 From: Jamezy Cesar Date: Mon, 22 Jun 2026 23:37:49 -0400 Subject: [PATCH] Tier 18.4: mapping CSV ROW_MISMATCH polish (post-audit followup #4 FINAL) Closes the last item on the post-audit followup list. With this merged, all four post-Tier-17.5 followups are complete and the cat-data-model is in a fully steady-state. Corrects 12 row-number references on fact_multileg_option_events rows in ddl/gold/06_cat_field_mapping.csv where the PDF-verified row number differs from the row originally claimed. All 12 corrections were identified during Tier 17.0's PDF verification pass and explicitly deferred to optional polish per Tier 17.5's CHANGELOG. Corrections in sec 5.2.2 (Multi-Leg Order Route Event): sender_imid: row 14 -> 13 destination: row 15 -> 14 destination_type: row 16 -> 15 routed_order_id: row 17 -> 16 session: row 18 -> 17 route_rejected_flag: row 27 -> 26 exch_origin_code: row 28 -> 27 paired_order_id: row 34 -> 28 Corrections in sec 5.2.6 (Multi-Leg Order Modified): prior_order_id: row 10 -> 9 initiator: row 22 -> 21 leaves_qty: row 27 -> 25 Corrections in sec 5.2.7 (Multi-Leg Order Cancelled): cancel_qty: row 13 -> 12 10 SECTION_MISMATCH cases identified in Tier 17.0 remain intentionally unmodified - they represent legitimate alternative references between multi-leg usage sites and single-leg canonical definition sites, both spec-supported. Verification: - All 8 guardrails pass. - 10/10 regression tests pass. - All 12 corrected rows match PDF spot-check exactly. POST-AUDIT STATE - STEADY: - F3.1 phantom backlog: 0 (closed Tier 13-16) - F3.2 missing-column backlog: 0 (closed Tier 17.1-17.5) - Post-audit tooling debt: 0 (closed Tier 18.1-18.4) - 4-dialect parity for option and multi-leg Gold layers - All 8 guardrails passing with 0 field-mapping allowlist exceptions --- CHANGELOG.md | 93 +++++++++++++++++++++++++++++++ ddl/gold/06_cat_field_mapping.csv | 24 ++++---- 2 files changed, 105 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1275b0e..158cdf2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,99 @@ All notable changes to the data model are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [Unreleased] - Tier 18.4: Mapping CSV ROW_MISMATCH polish (post-audit followup #4, FINAL) + +### Milestone + +**This sub-tier closes the last item on the post-audit followup list.** With this PR merged, all four post-Tier-17.5 followups are complete: +- ✅ #1 Validator regex hardening (Tier 18.1, PR #24) +- ✅ #2 Empty allowlist housekeeping + guardrails README (Tier 18.2, PR #25) +- ✅ #3 Multi-leg Fabric Lakehouse Gold variant (Tier 18.3, PR #26) +- ✅ #4 Mapping CSV section/row polish (this sub-tier) + +The cat-data-model is now in a fully steady-state. + +### Changed + +- `ddl/gold/06_cat_field_mapping.csv` - corrected 12 row-number references on `fact_multileg_option_events` rows where the PDF-verified row number differs from the row originally claimed in the mapping CSV. All 12 corrections were identified during Tier 17.0's PDF verification pass (recorded in `TIER_17_VERIFICATION.csv`) and deferred to optional polish per the Tier 17.5 CHANGELOG. + +### Corrections applied + +All in `fact_multileg_option_events`. Format: `column: §section old_row → new_row`. + +In §5.2.2 (Multi-Leg Order Route Event field table — row numbering differs slightly from equity §4.3): + +| Column | Old row | New row (PDF-verified, page) | +|---|---|---| +| `sender_imid` | 14 | 13 (p 291) | +| `destination` | 15 | 14 (p 291) | +| `destination_type` | 16 | 15 (p 291) | +| `routed_order_id` | 17 | 16 (p 291) | +| `session` | 18 | 17 (p 291) | +| `route_rejected_flag` | 27 | 26 (p 292) | +| `exch_origin_code` | 28 | 27 (p 292) | +| `paired_order_id` | 34 | 28 (p 292) | + +In §5.2.6 (Multi-Leg Order Modified field table): + +| Column | Old row | New row | +|---|---|---| +| `prior_order_id` | 10 | 9 (p 324) | +| `initiator` | 22 | 21 (p 325) | +| `leaves_qty` | 27 | 25 (p 325) | + +In §5.2.7 (Multi-Leg Order Cancelled field table): + +| Column | Old row | New row | +|---|---|---| +| `cancel_qty` | 13 | 12 (p 331) | + +### Why + +Cosmetic accuracy. These row numbers describe where each field appears in the multi-leg field-table rendering of CAT IM v4.1.0r15. The PDF verifier (`verify_cat_im_v3.py`) caught the off-by-one or off-by-multi-row discrepancies during Tier 17.0's comprehensive verification pass. The Tier 17.5 CHANGELOG explicitly deferred the polish: *"These references can be polished in an optional future tier (proposed Tier 18) without affecting DDL correctness or audit closure."* + +### Why this is the polish-stop point + +10 SECTION_MISMATCH cases identified in Tier 17.0 remain **intentionally unmodified**. They represent legitimate alternative references between: +- Multi-leg event usage sites (§5.2.x sections) — where the field appears in a specific multi-leg event's field table +- Single-leg canonical definition sites (§5.1.x sections) — where the field is first defined for option events + +Both reference styles are spec-supported. The mapping CSV's current §5.2.x style is meaningful in the multi-leg context and was preserved as a design decision. + +### Verification + +- All 8 guardrails pass. +- 10/10 regression tests in `test__ddl_parser.py` pass. +- All 12 corrected rows now match the PDF spot-check exactly. + +### Coverage + +``` +Validators: 8/8 pass +Allowlisted backlog: 0 (audit closed in Tier 17.5) +Field-mapping rows verified: 257 (unchanged) +DDL files in parity scope: 23 (unchanged) +Mapping CSV row corrections: 12 (this sub-tier) +``` + +### Post-followup state — STEADY + +All four post-audit followups are complete: + +| # | Item | PR | +|---|---|---| +| 1 | Validator regex hardening (balanced-paren `_ddl_parser.py`) | #24 | +| 2 | Empty allowlist housekeeping + guardrails README | #25 | +| 3 | Multi-leg Fabric Lakehouse Gold variant | #26 | +| 4 | Mapping CSV ROW_MISMATCH polish | this PR | + +The repository is in a fully steady-state: +- F3.1 phantom-table backlog: **0** (closed Tier 13-16) +- F3.2 missing-column backlog: **0** (closed Tier 17.1-17.5) +- Post-audit tooling debt: **0** (closed Tier 18.1-18.4) +- All 8 guardrails passing with 0 allowlist exceptions on field-mapping +- 4-dialect parity coverage for option and multi-leg Gold layers + ## [Unreleased] - Tier 18.3: Multi-leg Fabric Lakehouse Gold variant ### Added diff --git a/ddl/gold/06_cat_field_mapping.csv b/ddl/gold/06_cat_field_mapping.csv index 6124280..0461047 100644 --- a/ddl/gold/06_cat_field_mapping.csv +++ b/ddl/gold/06_cat_field_mapping.csv @@ -219,24 +219,24 @@ fact_multileg_option_events,representative_ind,representativeInd,OrderEvents,MLN fact_multileg_option_events,solicitation_flag,solicitationFlag,OrderEvents,MLNO,BOOLEAN,Y,1,Section 5.2.1 row 28: True if the order was originated in response to an RFQ or other solicitation,VERIFIED fact_multileg_option_events,rfq_id,RFQID,OrderEvents,"MLNO,MLOR,MLOA",STRING,N,64,Section 5.2.1 row 29: For events responding to an RFQ,VERIFIED fact_multileg_option_events,number_of_legs,numberOfLegs,OrderEvents,"MLNO,MLOR,MLOA",INT,Y,3,Section 5.2.1 row 30: Number of CAT-reportable legs in the multi-leg order,VERIFIED -fact_multileg_option_events,sender_imid,senderIMID,OrderEvents,"MLOR,MLMR,MLCR,MLOA,MLOM",STRING,N,8,Section 5.2.2 row 14: IMID of the IM sending the route,VERIFIED -fact_multileg_option_events,destination,destination,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,8,Section 5.2.2 row 15: Destination IMID,VERIFIED -fact_multileg_option_events,destination_type,destinationType,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,4,Section 5.2.2 row 16: Whether destination is exchange / affiliated firm / other,VERIFIED -fact_multileg_option_events,routed_order_id,routedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 17: ID assigned to the routed order,VERIFIED -fact_multileg_option_events,session,session,OrderEvents,"MLOR,MLMR",STRING,N,40,Section 5.2.2 row 18: Session ID used when routing,VERIFIED -fact_multileg_option_events,route_rejected_flag,routeRejectedFlag,OrderEvents,"MLOR,MLMR,MLCR",BOOLEAN,N,1,Section 5.2.2 row 27: True if the routed order was not accepted by destination,VERIFIED -fact_multileg_option_events,exch_origin_code,exchOriginCode,OrderEvents,MLOR,STRING,N,4,Section 5.2.2 row 28: Code signifying the origin of the account as sent to exchange,VERIFIED -fact_multileg_option_events,paired_order_id,pairedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 34 / 5.2.3 row 30: Paired order identifier,VERIFIED +fact_multileg_option_events,sender_imid,senderIMID,OrderEvents,"MLOR,MLMR,MLCR,MLOA,MLOM",STRING,N,8,Section 5.2.2 row 13: IMID of the IM sending the route,VERIFIED +fact_multileg_option_events,destination,destination,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,8,Section 5.2.2 row 14: Destination IMID,VERIFIED +fact_multileg_option_events,destination_type,destinationType,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,4,Section 5.2.2 row 15: Whether destination is exchange / affiliated firm / other,VERIFIED +fact_multileg_option_events,routed_order_id,routedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 16: ID assigned to the routed order,VERIFIED +fact_multileg_option_events,session,session,OrderEvents,"MLOR,MLMR",STRING,N,40,Section 5.2.2 row 17: Session ID used when routing,VERIFIED +fact_multileg_option_events,route_rejected_flag,routeRejectedFlag,OrderEvents,"MLOR,MLMR,MLCR",BOOLEAN,N,1,Section 5.2.2 row 26: True if the routed order was not accepted by destination,VERIFIED +fact_multileg_option_events,exch_origin_code,exchOriginCode,OrderEvents,MLOR,STRING,N,4,Section 5.2.2 row 27: Code signifying the origin of the account as sent to exchange,VERIFIED +fact_multileg_option_events,paired_order_id,pairedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 28 / 5.2.3 row 30: Paired order identifier,VERIFIED fact_multileg_option_events,receiver_imid,receiverIMID,OrderEvents,"MLOA,MLOM",STRING,N,8,Section 5.2.3 row 15: IMID of the IM receiving the order,VERIFIED fact_multileg_option_events,sender_type,senderType,OrderEvents,"MLOA,MLOM",STRING,Y,4,Section 5.2.3 row 17: Type of origin from which the order is routed,VERIFIED fact_multileg_option_events,originating_imid,originatingIMID,OrderEvents,"MLOR,MLOM,MLOC,MLOCR,MLOE",STRING,N,8,Section 5.2.2 row 9: Used in mergers/acquisitions where prior firm orders are still open,VERIFIED fact_multileg_option_events,prior_order_key_date,priorOrderKeyDate,OrderEvents,"MLOM,MLOE",DATETIME,N,30,Section 5.2.6 row 9: Prior orderKeyDate when a new orderID is assigned,VERIFIED -fact_multileg_option_events,prior_order_id,priorOrderID,OrderEvents,"MLOM,MLOE",STRING,N,64,Section 5.2.6 row 10: Prior orderID,VERIFIED +fact_multileg_option_events,prior_order_id,priorOrderID,OrderEvents,"MLOM,MLOE",STRING,N,64,Section 5.2.6 row 9: Prior orderID,VERIFIED fact_multileg_option_events,parent_order_key_date,parentOrderKeyDate,OrderEvents,MLCO,DATETIME,Y,30,Section 5.2.5.1 row 9: orderKeyDate of the parent event from which the Child Order was sliced,VERIFIED fact_multileg_option_events,parent_order_id,parentOrderID,OrderEvents,MLCO,STRING,Y,64,Section 5.2.5.1 row 10: orderID of the parent event,VERIFIED -fact_multileg_option_events,initiator,initiator,OrderEvents,"MLOM,MLOC,MLOCR",STRING,Y,4,Section 5.2.6 row 22: Indicates who initiated the modification/cancellation,VERIFIED -fact_multileg_option_events,leaves_qty,leavesQty,OrderEvents,"MLOM,MLOC",DECIMAL,N,15,Section 5.2.6 row 27 / 5.2.7 row 14: Number of units left open after modification or cancel,VERIFIED -fact_multileg_option_events,cancel_qty,cancelQty,OrderEvents,"MLOC,MLOCR",DECIMAL,N,15,Section 5.2.7 row 13: Quantity being cancelled,VERIFIED +fact_multileg_option_events,initiator,initiator,OrderEvents,"MLOM,MLOC,MLOCR",STRING,Y,4,Section 5.2.6 row 21: Indicates who initiated the modification/cancellation,VERIFIED +fact_multileg_option_events,leaves_qty,leavesQty,OrderEvents,"MLOM,MLOC",DECIMAL,N,15,Section 5.2.6 row 25 / 5.2.7 row 14: Number of units left open after modification or cancel,VERIFIED +fact_multileg_option_events,cancel_qty,cancelQty,OrderEvents,"MLOC,MLOCR",DECIMAL,N,15,Section 5.2.7 row 12: Quantity being cancelled,VERIFIED fact_multileg_option_events,request_timestamp,requestTimestamp,OrderEvents,"MLOM,MLOC",DATETIME,N,30,Section 5.2.6 row 33 / 5.2.7 row 17: Date/time the modification or cancellation was requested,VERIFIED fact_multileg_option_events,quote_key_date,quoteKeyDate,OrderEvents,"MLNQ,MLRQ,MLQS,MLQR,MLQC,MLQM",DATETIME,Y,30,Section 5.2.8 quote events: Date and time the quoteID was assigned,VERIFIED fact_multileg_option_events,quote_id,quoteID,OrderEvents,"MLNQ,MLRQ,MLQS,MLQR,MLQC,MLQM",STRING,Y,64,Section 5.2.8 quote events: Internal quote ID,VERIFIED