Skip to content

Tier 18.4: mapping CSV ROW_MISMATCH polish (post-audit followup #4 FINAL) #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jamezycesar-collab merged 1 commit into from
Jun 23, 2026
Merged

Tier 18.4: mapping CSV ROW_MISMATCH polish (post-audit followup #4 FINAL) #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,99 @@

All notable changes to the data model are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased] - Tier 18.4: Mapping CSV ROW_MISMATCH polish (post-audit followup #4, FINAL)

### Milestone

**This sub-tier closes the last item on the post-audit followup list.** With this PR merged, all four post-Tier-17.5 followups are complete:
- ✅ #1 Validator regex hardening (Tier 18.1, PR #24)
- ✅ #2 Empty allowlist housekeeping + guardrails README (Tier 18.2, PR #25)
- ✅ #3 Multi-leg Fabric Lakehouse Gold variant (Tier 18.3, PR #26)
- ✅ #4 Mapping CSV section/row polish (this sub-tier)

The cat-data-model is now in a fully steady-state.

### Changed

- `ddl/gold/06_cat_field_mapping.csv` - corrected 12 row-number references on `fact_multileg_option_events` rows where the PDF-verified row number differs from the row originally claimed in the mapping CSV. All 12 corrections were identified during Tier 17.0's PDF verification pass (recorded in `TIER_17_VERIFICATION.csv`) and deferred to optional polish per the Tier 17.5 CHANGELOG.

### Corrections applied

All in `fact_multileg_option_events`. Format: `column: §section old_row → new_row`.

In §5.2.2 (Multi-Leg Order Route Event field table — row numbering differs slightly from equity §4.3):

| Column | Old row | New row (PDF-verified, page) |
|---|---|---|
| `sender_imid` | 14 | 13 (p 291) |
| `destination` | 15 | 14 (p 291) |
| `destination_type` | 16 | 15 (p 291) |
| `routed_order_id` | 17 | 16 (p 291) |
| `session` | 18 | 17 (p 291) |
| `route_rejected_flag` | 27 | 26 (p 292) |
| `exch_origin_code` | 28 | 27 (p 292) |
| `paired_order_id` | 34 | 28 (p 292) |

In §5.2.6 (Multi-Leg Order Modified field table):

| Column | Old row | New row |
|---|---|---|
| `prior_order_id` | 10 | 9 (p 324) |
| `initiator` | 22 | 21 (p 325) |
| `leaves_qty` | 27 | 25 (p 325) |

In §5.2.7 (Multi-Leg Order Cancelled field table):

| Column | Old row | New row |
|---|---|---|
| `cancel_qty` | 13 | 12 (p 331) |

### Why

Cosmetic accuracy. These row numbers describe where each field appears in the multi-leg field-table rendering of CAT IM v4.1.0r15. The PDF verifier (`verify_cat_im_v3.py`) caught the off-by-one or off-by-multi-row discrepancies during Tier 17.0's comprehensive verification pass. The Tier 17.5 CHANGELOG explicitly deferred the polish: *"These references can be polished in an optional future tier (proposed Tier 18) without affecting DDL correctness or audit closure."*

### Why this is the polish-stop point

10 SECTION_MISMATCH cases identified in Tier 17.0 remain **intentionally unmodified**. They represent legitimate alternative references between:
- Multi-leg event usage sites (§5.2.x sections) — where the field appears in a specific multi-leg event's field table
- Single-leg canonical definition sites (§5.1.x sections) — where the field is first defined for option events

Both reference styles are spec-supported. The mapping CSV's current §5.2.x style is meaningful in the multi-leg context and was preserved as a design decision.

### Verification

- All 8 guardrails pass.
- 10/10 regression tests in `test__ddl_parser.py` pass.
- All 12 corrected rows now match the PDF spot-check exactly.

### Coverage

```
Validators: 8/8 pass
Allowlisted backlog: 0 (audit closed in Tier 17.5)
Field-mapping rows verified: 257 (unchanged)
DDL files in parity scope: 23 (unchanged)
Mapping CSV row corrections: 12 (this sub-tier)
```

### Post-followup state — STEADY

All four post-audit followups are complete:

| # | Item | PR |
|---|---|---|
| 1 | Validator regex hardening (balanced-paren `_ddl_parser.py`) | #24 |
| 2 | Empty allowlist housekeeping + guardrails README | #25 |
| 3 | Multi-leg Fabric Lakehouse Gold variant | #26 |
| 4 | Mapping CSV ROW_MISMATCH polish | this PR |

The repository is in a fully steady-state:
- F3.1 phantom-table backlog: **0** (closed Tier 13-16)
- F3.2 missing-column backlog: **0** (closed Tier 17.1-17.5)
- Post-audit tooling debt: **0** (closed Tier 18.1-18.4)
- All 8 guardrails passing with 0 allowlist exceptions on field-mapping
- 4-dialect parity coverage for option and multi-leg Gold layers

## [Unreleased] - Tier 18.3: Multi-leg Fabric Lakehouse Gold variant

### Added
Expand Down
24 changes: 12 additions & 12 deletions ddl/gold/06_cat_field_mapping.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,24 +219,24 @@ fact_multileg_option_events,representative_ind,representativeInd,OrderEvents,MLN
fact_multileg_option_events,solicitation_flag,solicitationFlag,OrderEvents,MLNO,BOOLEAN,Y,1,Section 5.2.1 row 28: True if the order was originated in response to an RFQ or other solicitation,VERIFIED
fact_multileg_option_events,rfq_id,RFQID,OrderEvents,"MLNO,MLOR,MLOA",STRING,N,64,Section 5.2.1 row 29: For events responding to an RFQ,VERIFIED
fact_multileg_option_events,number_of_legs,numberOfLegs,OrderEvents,"MLNO,MLOR,MLOA",INT,Y,3,Section 5.2.1 row 30: Number of CAT-reportable legs in the multi-leg order,VERIFIED
fact_multileg_option_events,sender_imid,senderIMID,OrderEvents,"MLOR,MLMR,MLCR,MLOA,MLOM",STRING,N,8,Section 5.2.2 row 14: IMID of the IM sending the route,VERIFIED
fact_multileg_option_events,destination,destination,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,8,Section 5.2.2 row 15: Destination IMID,VERIFIED
fact_multileg_option_events,destination_type,destinationType,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,4,Section 5.2.2 row 16: Whether destination is exchange / affiliated firm / other,VERIFIED
fact_multileg_option_events,routed_order_id,routedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 17: ID assigned to the routed order,VERIFIED
fact_multileg_option_events,session,session,OrderEvents,"MLOR,MLMR",STRING,N,40,Section 5.2.2 row 18: Session ID used when routing,VERIFIED
fact_multileg_option_events,route_rejected_flag,routeRejectedFlag,OrderEvents,"MLOR,MLMR,MLCR",BOOLEAN,N,1,Section 5.2.2 row 27: True if the routed order was not accepted by destination,VERIFIED
fact_multileg_option_events,exch_origin_code,exchOriginCode,OrderEvents,MLOR,STRING,N,4,Section 5.2.2 row 28: Code signifying the origin of the account as sent to exchange,VERIFIED
fact_multileg_option_events,paired_order_id,pairedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 34 / 5.2.3 row 30: Paired order identifier,VERIFIED
fact_multileg_option_events,sender_imid,senderIMID,OrderEvents,"MLOR,MLMR,MLCR,MLOA,MLOM",STRING,N,8,Section 5.2.2 row 13: IMID of the IM sending the route,VERIFIED
fact_multileg_option_events,destination,destination,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,8,Section 5.2.2 row 14: Destination IMID,VERIFIED
fact_multileg_option_events,destination_type,destinationType,OrderEvents,"MLOR,MLMR,MLCR",STRING,Y,4,Section 5.2.2 row 15: Whether destination is exchange / affiliated firm / other,VERIFIED
fact_multileg_option_events,routed_order_id,routedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 16: ID assigned to the routed order,VERIFIED
fact_multileg_option_events,session,session,OrderEvents,"MLOR,MLMR",STRING,N,40,Section 5.2.2 row 17: Session ID used when routing,VERIFIED
fact_multileg_option_events,route_rejected_flag,routeRejectedFlag,OrderEvents,"MLOR,MLMR,MLCR",BOOLEAN,N,1,Section 5.2.2 row 26: True if the routed order was not accepted by destination,VERIFIED
fact_multileg_option_events,exch_origin_code,exchOriginCode,OrderEvents,MLOR,STRING,N,4,Section 5.2.2 row 27: Code signifying the origin of the account as sent to exchange,VERIFIED
fact_multileg_option_events,paired_order_id,pairedOrderID,OrderEvents,"MLOR,MLOA",STRING,N,64,Section 5.2.2 row 28 / 5.2.3 row 30: Paired order identifier,VERIFIED
fact_multileg_option_events,receiver_imid,receiverIMID,OrderEvents,"MLOA,MLOM",STRING,N,8,Section 5.2.3 row 15: IMID of the IM receiving the order,VERIFIED
fact_multileg_option_events,sender_type,senderType,OrderEvents,"MLOA,MLOM",STRING,Y,4,Section 5.2.3 row 17: Type of origin from which the order is routed,VERIFIED
fact_multileg_option_events,originating_imid,originatingIMID,OrderEvents,"MLOR,MLOM,MLOC,MLOCR,MLOE",STRING,N,8,Section 5.2.2 row 9: Used in mergers/acquisitions where prior firm orders are still open,VERIFIED
fact_multileg_option_events,prior_order_key_date,priorOrderKeyDate,OrderEvents,"MLOM,MLOE",DATETIME,N,30,Section 5.2.6 row 9: Prior orderKeyDate when a new orderID is assigned,VERIFIED
fact_multileg_option_events,prior_order_id,priorOrderID,OrderEvents,"MLOM,MLOE",STRING,N,64,Section 5.2.6 row 10: Prior orderID,VERIFIED
fact_multileg_option_events,prior_order_id,priorOrderID,OrderEvents,"MLOM,MLOE",STRING,N,64,Section 5.2.6 row 9: Prior orderID,VERIFIED
fact_multileg_option_events,parent_order_key_date,parentOrderKeyDate,OrderEvents,MLCO,DATETIME,Y,30,Section 5.2.5.1 row 9: orderKeyDate of the parent event from which the Child Order was sliced,VERIFIED
fact_multileg_option_events,parent_order_id,parentOrderID,OrderEvents,MLCO,STRING,Y,64,Section 5.2.5.1 row 10: orderID of the parent event,VERIFIED
fact_multileg_option_events,initiator,initiator,OrderEvents,"MLOM,MLOC,MLOCR",STRING,Y,4,Section 5.2.6 row 22: Indicates who initiated the modification/cancellation,VERIFIED
fact_multileg_option_events,leaves_qty,leavesQty,OrderEvents,"MLOM,MLOC",DECIMAL,N,15,Section 5.2.6 row 27 / 5.2.7 row 14: Number of units left open after modification or cancel,VERIFIED
fact_multileg_option_events,cancel_qty,cancelQty,OrderEvents,"MLOC,MLOCR",DECIMAL,N,15,Section 5.2.7 row 13: Quantity being cancelled,VERIFIED
fact_multileg_option_events,initiator,initiator,OrderEvents,"MLOM,MLOC,MLOCR",STRING,Y,4,Section 5.2.6 row 21: Indicates who initiated the modification/cancellation,VERIFIED
fact_multileg_option_events,leaves_qty,leavesQty,OrderEvents,"MLOM,MLOC",DECIMAL,N,15,Section 5.2.6 row 25 / 5.2.7 row 14: Number of units left open after modification or cancel,VERIFIED
fact_multileg_option_events,cancel_qty,cancelQty,OrderEvents,"MLOC,MLOCR",DECIMAL,N,15,Section 5.2.7 row 12: Quantity being cancelled,VERIFIED
fact_multileg_option_events,request_timestamp,requestTimestamp,OrderEvents,"MLOM,MLOC",DATETIME,N,30,Section 5.2.6 row 33 / 5.2.7 row 17: Date/time the modification or cancellation was requested,VERIFIED
fact_multileg_option_events,quote_key_date,quoteKeyDate,OrderEvents,"MLNQ,MLRQ,MLQS,MLQR,MLQC,MLQM",DATETIME,Y,30,Section 5.2.8 quote events: Date and time the quoteID was assigned,VERIFIED
fact_multileg_option_events,quote_id,quoteID,OrderEvents,"MLNQ,MLRQ,MLQS,MLQR,MLQC,MLQM",STRING,Y,64,Section 5.2.8 quote events: Internal quote ID,VERIFIED
Expand Down
Loading