diff --git a/CHANGELOG.md b/CHANGELOG.md index 1275b0e..158cdf2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,99 @@ All notable changes to the data model are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [Unreleased] - Tier 18.4: Mapping CSV ROW_MISMATCH polish (post-audit followup #4, FINAL) + +### Milestone + +**This sub-tier closes the last item on the post-audit followup list.** With this PR merged, all four post-Tier-17.5 followups are complete: +- ✅ #1 Validator regex hardening (Tier 18.1, PR #24) +- ✅ #2 Empty allowlist housekeeping + guardrails README (Tier 18.2, PR #25) +- ✅ #3 Multi-leg Fabric Lakehouse Gold variant (Tier 18.3, PR #26) +- ✅ #4 Mapping CSV section/row polish (this sub-tier) + +The cat-data-model is now in a fully steady-state. + +### Changed + +- `ddl/gold/06_cat_field_mapping.csv` - corrected 12 row-number references on `fact_multileg_option_events` rows where the PDF-verified row number differs from the row originally claimed in the mapping CSV. All 12 corrections were identified during Tier 17.0's PDF verification pass (recorded in `TIER_17_VERIFICATION.csv`) and deferred to optional polish per the Tier 17.5 CHANGELOG. + +### Corrections applied + +All in `fact_multileg_option_events`. Format: `column: §section old_row → new_row`. + +In §5.2.2 (Multi-Leg Order Route Event field table — row numbering differs slightly from equity §4.3): + +| Column | Old row | New row (PDF-verified, page) | +|---|---|---| +| `sender_imid` | 14 | 13 (p 291) | +| `destination` | 15 | 14 (p 291) | +| `destination_type` | 16 | 15 (p 291) | +| `routed_order_id` | 17 | 16 (p 291) | +| `session` | 18 | 17 (p 291) | +| `route_rejected_flag` | 27 | 26 (p 292) | +| `exch_origin_code` | 28 | 27 (p 292) | +| `paired_order_id` | 34 | 28 (p 292) | + +In §5.2.6 (Multi-Leg Order Modified field table): + +| Column | Old row | New row | +|---|---|---| +| `prior_order_id` | 10 | 9 (p 324) | +| `initiator` | 22 | 21 (p 325) | +| `leaves_qty` | 27 | 25 (p 325) | + +In §5.2.7 (Multi-Leg Order Cancelled field table): + +| Column | Old row | New row | +|---|---|---| +| `cancel_qty` | 13 | 12 (p 331) | + +### Why + +Cosmetic accuracy. These row numbers describe where each field appears in the multi-leg field-table rendering of CAT IM v4.1.0r15. The PDF verifier (`verify_cat_im_v3.py`) caught the off-by-one or off-by-multi-row discrepancies during Tier 17.0's comprehensive verification pass. The Tier 17.5 CHANGELOG explicitly deferred the polish: *"These references can be polished in an optional future tier (proposed Tier 18) without affecting DDL correctness or audit closure."* + +### Why this is the polish-stop point + +10 SECTION_MISMATCH cases identified in Tier 17.0 remain **intentionally unmodified**. They represent legitimate alternative references between: +- Multi-leg event usage sites (§5.2.x sections) — where the field appears in a specific multi-leg event's field table +- Single-leg canonical definition sites (§5.1.x sections) — where the field is first defined for option events + +Both reference styles are spec-supported. The mapping CSV's current §5.2.x style is meaningful in the multi-leg context and was preserved as a design decision. + +### Verification + +- All 8 guardrails pass. +- 10/10 regression tests in `test__ddl_parser.py` pass. +- All 12 corrected rows now match the PDF spot-check exactly. + +### Coverage + +``` +Validators: 8/8 pass +Allowlisted backlog: 0 (audit closed in Tier 17.5) +Field-mapping rows verified: 257 (unchanged) +DDL files in parity scope: 23 (unchanged) +Mapping CSV row corrections: 12 (this sub-tier) +``` + +### Post-followup state — STEADY + +All four post-audit followups are complete: + +| # | Item | PR | +|---|---|---| +| 1 | Validator regex hardening (balanced-paren `_ddl_parser.py`) | #24 | +| 2 | Empty allowlist housekeeping + guardrails README | #25 | +| 3 | Multi-leg Fabric Lakehouse Gold variant | #26 | +| 4 | Mapping CSV ROW_MISMATCH polish | this PR | + +The repository is in a fully steady-state: +- F3.1 phantom-table backlog: **0** (closed Tier 13-16) +- F3.2 missing-column backlog: **0** (closed Tier 17.1-17.5) +- Post-audit tooling debt: **0** (closed Tier 18.1-18.4) +- All 8 guardrails passing with 0 allowlist exceptions on field-mapping +- 4-dialect parity coverage for option and multi-leg Gold layers + ## [Unreleased] - Tier 18.3: Multi-leg Fabric Lakehouse Gold variant ### Added diff --git a/ddl/gold/06_cat_field_mapping.csv b/ddl/gold/06_cat_field_mapping.csv index 6124280..0461047 100644 --- a/ddl/gold/06_cat_field_mapping.csv +++ b/ddl/gold/06_cat_field_mapping.csv @@ -219,24 +219,24 @@ fact_multileg_option_events,representative_ind,representativeInd,OrderEvents,MLN fact_multileg_option_events,solicitation_flag,solicitationFlag,OrderEvents,MLNO,BOOLEAN,Y,1,Section 5.2.1 row 28: True if the order was originated in response to an RFQ or other solicitation,VERIFIED fact_multileg_option_events,rfq_id,RFQID,OrderEvents,"MLNO,MLOR,MLOA",STRING,N,64,Section 5.2.1 row 29: For events responding to an RFQ,VERIFIED fact_multileg_option_events,number_of_legs,numberOfLegs,OrderEvents,"MLNO,MLOR,MLOA",INT,Y,3,Section 5.2.1 row 30: Number of CAT-reportable legs in the multi-leg order,VERIFIED -fact_multileg_option_events,sender_imid,senderIMID,OrderEvents,"MLOR,MLMR,MLCR,MLOA,MLOM",STRING,N,8,Section 5.2.2 row 14: IMID of the IM sending the route,VERIFIED -fact_multileg_option_events,destination,destination,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,8,Section 5.2.2 row 15: Destination IMID,VERIFIED -fact_multileg_option_events,destination_type,destinationType,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,4,Section 5.2.2 row 16: Whether destination is exchange / affiliated firm / other,VERIFIED -fact_multileg_option_events,routed_order_id,routedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 17: ID assigned to the routed order,VERIFIED -fact_multileg_option_events,session,session,OrderEvents,"MLOR,MLMR",STRING,N,40,Section 5.2.2 row 18: Session ID used when routing,VERIFIED -fact_multileg_option_events,route_rejected_flag,routeRejectedFlag,OrderEvents,"MLOR,MLMR,MLCR",BOOLEAN,N,1,Section 5.2.2 row 27: True if the routed order was not accepted by destination,VERIFIED -fact_multileg_option_events,exch_origin_code,exchOriginCode,OrderEvents,MLOR,STRING,N,4,Section 5.2.2 row 28: Code signifying the origin of the account as sent to exchange,VERIFIED -fact_multileg_option_events,paired_order_id,pairedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 34 / 5.2.3 row 30: Paired order identifier,VERIFIED +fact_multileg_option_events,sender_imid,senderIMID,OrderEvents,"MLOR,MLMR,MLCR,MLOA,MLOM",STRING,N,8,Section 5.2.2 row 13: IMID of the IM sending the route,VERIFIED +fact_multileg_option_events,destination,destination,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,8,Section 5.2.2 row 14: Destination IMID,VERIFIED +fact_multileg_option_events,destination_type,destinationType,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,4,Section 5.2.2 row 15: Whether destination is exchange / affiliated firm / other,VERIFIED +fact_multileg_option_events,routed_order_id,routedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 16: ID assigned to the routed order,VERIFIED +fact_multileg_option_events,session,session,OrderEvents,"MLOR,MLMR",STRING,N,40,Section 5.2.2 row 17: Session ID used when routing,VERIFIED +fact_multileg_option_events,route_rejected_flag,routeRejectedFlag,OrderEvents,"MLOR,MLMR,MLCR",BOOLEAN,N,1,Section 5.2.2 row 26: True if the routed order was not accepted by destination,VERIFIED +fact_multileg_option_events,exch_origin_code,exchOriginCode,OrderEvents,MLOR,STRING,N,4,Section 5.2.2 row 27: Code signifying the origin of the account as sent to exchange,VERIFIED +fact_multileg_option_events,paired_order_id,pairedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 28 / 5.2.3 row 30: Paired order identifier,VERIFIED fact_multileg_option_events,receiver_imid,receiverIMID,OrderEvents,"MLOA,MLOM",STRING,N,8,Section 5.2.3 row 15: IMID of the IM receiving the order,VERIFIED fact_multileg_option_events,sender_type,senderType,OrderEvents,"MLOA,MLOM",STRING,Y,4,Section 5.2.3 row 17: Type of origin from which the order is routed,VERIFIED fact_multileg_option_events,originating_imid,originatingIMID,OrderEvents,"MLOR,MLOM,MLOC,MLOCR,MLOE",STRING,N,8,Section 5.2.2 row 9: Used in mergers/acquisitions where prior firm orders are still open,VERIFIED fact_multileg_option_events,prior_order_key_date,priorOrderKeyDate,OrderEvents,"MLOM,MLOE",DATETIME,N,30,Section 5.2.6 row 9: Prior orderKeyDate when a new orderID is assigned,VERIFIED -fact_multileg_option_events,prior_order_id,priorOrderID,OrderEvents,"MLOM,MLOE",STRING,N,64,Section 5.2.6 row 10: Prior orderID,VERIFIED +fact_multileg_option_events,prior_order_id,priorOrderID,OrderEvents,"MLOM,MLOE",STRING,N,64,Section 5.2.6 row 9: Prior orderID,VERIFIED fact_multileg_option_events,parent_order_key_date,parentOrderKeyDate,OrderEvents,MLCO,DATETIME,Y,30,Section 5.2.5.1 row 9: orderKeyDate of the parent event from which the Child Order was sliced,VERIFIED fact_multileg_option_events,parent_order_id,parentOrderID,OrderEvents,MLCO,STRING,Y,64,Section 5.2.5.1 row 10: orderID of the parent event,VERIFIED -fact_multileg_option_events,initiator,initiator,OrderEvents,"MLOM,MLOC,MLOCR",STRING,Y,4,Section 5.2.6 row 22: Indicates who initiated the modification/cancellation,VERIFIED -fact_multileg_option_events,leaves_qty,leavesQty,OrderEvents,"MLOM,MLOC",DECIMAL,N,15,Section 5.2.6 row 27 / 5.2.7 row 14: Number of units left open after modification or cancel,VERIFIED -fact_multileg_option_events,cancel_qty,cancelQty,OrderEvents,"MLOC,MLOCR",DECIMAL,N,15,Section 5.2.7 row 13: Quantity being cancelled,VERIFIED +fact_multileg_option_events,initiator,initiator,OrderEvents,"MLOM,MLOC,MLOCR",STRING,Y,4,Section 5.2.6 row 21: Indicates who initiated the modification/cancellation,VERIFIED +fact_multileg_option_events,leaves_qty,leavesQty,OrderEvents,"MLOM,MLOC",DECIMAL,N,15,Section 5.2.6 row 25 / 5.2.7 row 14: Number of units left open after modification or cancel,VERIFIED +fact_multileg_option_events,cancel_qty,cancelQty,OrderEvents,"MLOC,MLOCR",DECIMAL,N,15,Section 5.2.7 row 12: Quantity being cancelled,VERIFIED fact_multileg_option_events,request_timestamp,requestTimestamp,OrderEvents,"MLOM,MLOC",DATETIME,N,30,Section 5.2.6 row 33 / 5.2.7 row 17: Date/time the modification or cancellation was requested,VERIFIED fact_multileg_option_events,quote_key_date,quoteKeyDate,OrderEvents,"MLNQ,MLRQ,MLQS,MLQR,MLQC,MLQM",DATETIME,Y,30,Section 5.2.8 quote events: Date and time the quoteID was assigned,VERIFIED fact_multileg_option_events,quote_id,quoteID,OrderEvents,"MLNQ,MLRQ,MLQS,MLQR,MLQC,MLQM",STRING,Y,64,Section 5.2.8 quote events: Internal quote ID,VERIFIED