fix: scope schema-drift check to litigation cases - #84
Conversation
check_schema_drift counted blank case_type across the entire court_cases table with an absolute >10 threshold, which nightly false-failed the supreme/special CronJobs (whose data is complete). The ~20k blanks are all non-litigation registrations (marriage, power-of-attorney, guardianship, etc.) filed under a numeric case-number namespace that legitimately has no case_type. Restrict the check to litigation rows (alphabetic case-number codes), fail on a ratio instead of an absolute count so it scales with table size, and allow optional per-court scoping via argv. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 28 minutes and 45 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses rolling per-developer review limits. Reviews become available again as older review attempts age out of the rolling limit window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the schema drift check script to calculate a ratio of missing case types specifically for litigation cases, rather than using an absolute count across all cases. It also adds support for scoping the check to specific court identifiers. The review feedback highlights two key issues: first, calculating the drift ratio over the entire table history can dilute recent failures and cause performance bottlenecks, so it is recommended to scope the check to recent cases (e.g., the last 3 days); second, providing invalid court identifiers currently results in a silent skip rather than an error, which should be resolved by validating the identifiers beforehand.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| where = [LITIGATION_ONLY] | ||
| params = {} | ||
| if courts: | ||
| where.append("court_identifier = ANY(:courts)") | ||
| params["courts"] = list(courts) | ||
| scope = " AND ".join(where) |
There was a problem hiding this comment.
⚠️ High: Drift Ratio Dilution & Performance Bottleneck
Calculating the schema drift ratio over the entire history of the court_cases table (1.5M+ rows) introduces two major issues:
- Dilution of Drift (False Pass): If the scraper starts failing today (e.g., 100% of today's 1,000 new cases are missing
case_type), the ratio of missing cases over the entire 1.5M historical rows will be1,000 / 1,501,000 ≈ 0.06%. This is far below the2%threshold (DRIFT_RATIO_THRESHOLD), meaning the check will pass and the drift will go undetected. - Performance: Evaluating the regex
split_part(case_number, '-', 2) ~ '[A-Za-z]'via a full table scan on 1.5M+ rows on every cron job run is extremely slow and does not scale.
Recommendation
Scope the check to recently created cases (e.g., the last 3 days). This ensures that a sudden scraper failure on a new batch will immediately spike the ratio to ~100% and trigger the alert, while also keeping the query highly performant.
| where = [LITIGATION_ONLY] | |
| params = {} | |
| if courts: | |
| where.append("court_identifier = ANY(:courts)") | |
| params["courts"] = list(courts) | |
| scope = " AND ".join(where) | |
| where = [LITIGATION_ONLY, "created_at >= NOW() - INTERVAL '3 days'"] | |
| params = {} | |
| if courts: | |
| where.append("court_identifier = ANY(:courts)") | |
| params["courts"] = list(courts) | |
| scope = " AND ".join(where) |
| def check(courts=None): | ||
| db_url = os.getenv("DATABASE_URL") | ||
| if not db_url: | ||
| print("DATABASE_URL not set") | ||
| sys.exit(1) |
There was a problem hiding this comment.
⚠️ Medium: Silent Skip on Invalid Court Identifiers
If a user passes an invalid court identifier (e.g., due to a typo like supremee), the script currently queries court_identifier = ANY(['supremee']). Since no rows match, total will be 0, and the script will print:
Schema check skipped: no litigation rows in scope (courts=['supremee'])
and exit with code 0 (success).
This silently skips the schema check instead of alerting the user that they provided an invalid court name.
Recommendation
Validate the provided court identifiers using the existing is_valid_court_identifier utility before running the query.
| def check(courts=None): | |
| db_url = os.getenv("DATABASE_URL") | |
| if not db_url: | |
| print("DATABASE_URL not set") | |
| sys.exit(1) | |
| def check(courts=None): | |
| db_url = os.getenv("DATABASE_URL") | |
| if not db_url: | |
| print("DATABASE_URL not set") | |
| sys.exit(1) | |
| if courts: | |
| from ngm.utils.court_mapping import is_valid_court_identifier | |
| invalid = [c for c in courts if not is_valid_court_identifier(c)] | |
| if invalid: | |
| print(f"ERROR: Invalid court identifier(s): {invalid}") | |
| sys.exit(1) |
…e courts - Fix CI: black formatting. - Only evaluate rows ingested in the last 3 days so a fresh selector break isn't diluted into insignificance by table history (and avoid scanning millions of rows each run); skip when the recent sample is below a floor. - Validate court identifiers passed via argv and error on unknown ones, rather than silently skipping (which would mask a typo as a pass). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Addressed the review feedback and fixed CI:
|
1 participant
Summary
The
ngm-supreme-courtandngm-special-courtCronJobs have been failing on every run for days. Both scrape complete data (0 missingcase_type), but they're killed byscripts/check_schema_drift.py, which sits in the&&chain between scrape and enrichment.The guard counts blank
case_typeacross the entire 1.56M-rowcourt_casestable with an absolute threshold of>10, then aborts. The ~20,368 blanks it trips on are all non-litigation administrative registrations (marriage-by-registration, power-of-attorney, guardianship, adoption, death-declaration) — filed under a numeric case-number namespace (e.g.082-02-0001) that legitimately carries nocase_type. Litigation cases use an alphabetic namespace (CP,FN,CR,WO,C1, …) and have zero blanks.Verified split against prod:
Changes
>10, so the check scales with table size and catches genuine selector drift.check_schema_drift.py supreme special).Test plan
python -m py_compile+ruff checkpasscase_type(guard now passes)ngm-supreme-court/ngm-special-courtruns complete green🤖 Generated with Claude Code