Skip to content

Tier 17.4: F3.2 sub-tier 4 — fact_option_order_events columns (30 cols) - #22

Merged
jamezycesar-collab merged 1 commit into
mainfrom
tier17-4-f32-fact-option-order-events
Jun 18, 2026
Merged

Tier 17.4: F3.2 sub-tier 4 — fact_option_order_events columns (30 cols)#22
jamezycesar-collab merged 1 commit into
mainfrom
tier17-4-f32-fact-option-order-events

Conversation

@jamezycesar-collab

Copy link
Copy Markdown
Owner

Summary

Fourth sub-tier of WS2 F3.2 follow-on. Adds 30 spec-mapping columns to fact_option_order_events across all four dialects.

Files changed

  • ddl/option/02_option_gold_delta.sql - 30 columns added
  • ddl/option/04_option_gold_hive.sql - 30 columns added (Hive line comments to avoid validator COMMENT terminator)
  • ddl/option/05_option_silver_fabric_lakehouse.sql - 30 columns added
  • ddl/option/06_option_fabric_warehouse.sql - 30 columns added
  • guardrails/known_field_mapping_gaps.csv - 81 → 51 rows
  • CHANGELOG.md - Tier 17.4 entry

30 columns added (PDF-verified)

Group Columns Spec section
Order key + ts order_key_date, event_timestamp §5.1.1 rows 6, 9
Manual flags manual_flag, manual_order_key_date, manual_order_id, electronic_dup_flag, electronic_timestamp §5.1.1 rows 10-14
Order metadata dept_type, min_qty, solicitation_flag, rfq_id, representative_ind, exch_origin_code §5.1.1 rows 15, 19, 28, 30, 32 + §5.1.3 row 28
Party / account firm_designated_id, account_holder_type, affiliate_flag §5.1.1 rows 24-26
Routing sender_imid, receiver_imid, sender_type, originating_imid, destination, destination_type, routed_order_id, session, route_rejected_flag, multi_leg_ind, paired_order_id §5.1.3 rows 9, 14-18, 27, 30, 34 + §5.1.4 rows 15, 17
Modify / parent prior_order_key_date, parent_order_key_date, request_timestamp §5.1.6.1 row 9, §5.1.7 rows 9, 33

No mapping CSV fixes

All 30 columns passed PDF verification with OK status (verifier output in TIER_17_VERIFICATION.csv). The mapping CSV's section refs are already correct.

Validator quirk found and worked around

The validator's _CREATE_TABLE_RE regex includes COMMENT in its terminator alternation. When the Hive variant initially used per-column COMMENT 'section ...' documentation, the regex misinterpreted the first column-level COMMENT appearing after a DECIMAL(38, 18)) close-paren as the table-level terminator — silently truncating column discovery and triggering false parity errors.

Workaround in this PR: Hive uses -- section ... line comments instead of inline COMMENT 'string'. Delta / Fabric Lakehouse / Fabric Warehouse variants use -- end-of-line comments (their conventional style anyway).

Followup (tooling debt): harden the validator's regex to require column-list close-paren before considering COMMENT as a body terminator. Tracking outside this PR.

Coverage

All 8 guardrails:                  pass (exit 0)
known_field_mapping_gaps.csv:    51 (was 81; -30)
  F3.1 phantom-table rows:        0 (unchanged - cleared in Tier 13-16)
  F3.2 missing-column rows:      51 (was 81)
DDL files in parity scope:        22 (unchanged)
Tables in 2+ dialects:            42 (unchanged - same tables, new columns)
New parity violations:             0

F3.2 burndown progress

Sub-tier Host table Cols Allowlist Status
Tier 17.1 fact_option_executions 3 92 → 89
Tier 17.2 fact_option_allocations 3 89 → 86
Tier 17.3 fact_multileg_option_legs 5 86 → 81
Tier 17.4 (this) fact_option_order_events 30 81 → 51
Tier 17.5 fact_multileg_option_events 51 51 → 0 next (final)

Test plan

  • All 8 guardrails pass (exit 0)
  • No new parity violations
  • All 30 fact_option_order_events mapping rows now resolve in DDL
  • PDF verification: all 30 cols match mapping CSV section + row
  • Recurrence-safe precheck in push script

… (30 cols)

Fourth sub-tier of WS2 F3.2 follow-on. Largest single addition
in the F3.2 closure: 30 spec-mapping columns added to
gold.fact_option_order_events across all four dialects (Delta,
Hive, Fabric Lakehouse, Fabric Warehouse).

Column groups:
  - Order key + timestamps (2): order_key_date, event_timestamp
  - Manual event flags (5): manual_flag, manual_order_key_date,
    manual_order_id, electronic_dup_flag, electronic_timestamp
  - Order metadata (6): dept_type, min_qty, solicitation_flag,
    rfq_id, representative_ind, exch_origin_code
  - Party / account (3): firm_designated_id, account_holder_type,
    affiliate_flag
  - Routing (11): sender_imid, receiver_imid, sender_type,
    originating_imid, destination, destination_type,
    routed_order_id, session, route_rejected_flag, multi_leg_ind,
    paired_order_id
  - Modify / parent linkage (3): prior_order_key_date,
    parent_order_key_date, request_timestamp

All 30 columns PDF-verified against CAT IM v4.1.0r15 sections
5.1.1, 5.1.3, 5.1.4, 5.1.6.1, and 5.1.7. No mapping CSV
section-ref corrections needed (all 30 passed verification
with OK status).

Mirrors the equity fact_order_events structure introduced in
Tier 16 (30 cols here is a subset of the 59 in equity equivalent).

Lesson-learned (tooling debt tracked separately): Hive variant
uses '-- section ...' line comments rather than per-column
COMMENT 'string' clauses, because the validator's CREATE TABLE
regex treats COMMENT as a body terminator and silently truncated
column discovery on first attempt.

Allowlist:
  known_field_mapping_gaps.csv: 81 -> 51 rows (-30)
  F3.2 remaining: 51 (was 81; only fact_multileg_option_events left)
  F3.1 phantom-table: 0 (unchanged, cleared in Tier 13-16)

All 8 guardrails pass.
@jamezycesar-collab
jamezycesar-collab merged commit f0f1e92 into main Jun 18, 2026
1 check passed
@jamezycesar-collab
jamezycesar-collab deleted the tier17-4-f32-fact-option-order-events branch June 18, 2026 04:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant