fix: backfill district case_type from detail page - #85
Conversation
District case_type was only ever taken from the daily cause-list listing's "मुद्दा विषय" (subject) column, which is blank for many administrative filings and which enrichment never backfilled — unlike high/supreme courts, whose enrichment reads "मुद्दाको किसिम" from the detail page (hence 0 nulls). This left ~20k district rows with no case_type. The detail page does expose the real "मुद्दाको किसिम". Map it to case_type during enrichment, and revisit already-enriched rows with a blank case_type to backfill them — using a dedicated path that only sets case_type and does not rebuild entities/hearings (whose delete-and-recreate risks downstream linkages). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 28 minutes and 35 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses rolling per-developer review limits. Reviews become available again as older review attempts age out of the rolling limit window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to backfill the case_type field for already-enriched court cases where it is missing, querying these cases in start_requests and updating them without rebuilding related entities. The reviewer recommends combining the status check and backfill logic within the existing transaction block to avoid redundant database queries. Additionally, they suggest setting a placeholder value (such as "-") when the case_type is missing on the detail page to prevent infinite re-scraping loops, which would also allow for the removal of the redundant _backfill_case_type helper method.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| already_enriched = case.status == "enriched" | ||
| has_case_type = bool(case.case_type) | ||
|
|
||
| if already_enriched: | ||
| if has_case_type: | ||
| self.logger.info(f"Case {case_number} already enriched, skipping") | ||
| return | ||
| # Already enriched but missing case_type: backfill just that field | ||
| # from the detail page without re-touching entities/hearings (whose | ||
| # rebuild deletes rows that may have downstream linkages). | ||
| case_type = self._extract_enrichment_data(soup).get("case_type") | ||
| if case_type: | ||
| self._backfill_case_type(case_number, code_name, case_type) | ||
| self.logger.info( | ||
| f"Backfilled case_type for {case_number} ({code_name}): {case_type}" | ||
| ) | ||
| else: | ||
| self.logger.info( | ||
| f"Case {case_number} ({code_name}) has no case_type on detail page" | ||
| ) | ||
| return |
There was a problem hiding this comment.
Combining the status check and the backfill logic inside the existing transaction block avoids querying the database twice for the same CourtCase row and opening/closing multiple transactions.
Additionally, if a case is already enriched but has no case_type on the detail page, leaving it empty will cause it to be repeatedly queried and scraped on every subsequent run of the spider (due to the query in start_requests). Setting a placeholder like "-" prevents this infinite re-scraping loop.
if case.status == "enriched":
if case.case_type:
self.logger.info(f"Case {case_number} already enriched, skipping")
return
# Already enriched but missing case_type: backfill just that field
# from the detail page without re-touching entities/hearings (whose
# rebuild deletes rows that may have downstream linkages).
case_type = self._extract_enrichment_data(soup).get("case_type")
if case_type:
case.case_type = case_type[:200]
case.updated_at = datetime.now(KATHMANDU_TZ).replace(tzinfo=None)
self.logger.info(
f"Backfilled case_type for {case_number} ({code_name}): {case_type}"
)
else:
# Set a placeholder to prevent infinite re-scraping in start_requests
case.case_type = "-"
case.updated_at = datetime.now(KATHMANDU_TZ).replace(tzinfo=None)
self.logger.info(
f"Case {case_number} ({code_name}) has no case_type on detail page; set to '-'"
)
return| def _backfill_case_type(self, case_number: str, code_name: str, case_type: str): | ||
| """Set case_type on an already-enriched case, leaving everything else.""" | ||
| now = datetime.now(KATHMANDU_TZ).replace(tzinfo=None) | ||
|
|
||
| with self.session.begin(): | ||
| case = ( | ||
| self.session.query(CourtCase) | ||
| .filter( | ||
| and_( | ||
| CourtCase.case_number == case_number, | ||
| CourtCase.court_identifier == code_name, | ||
| ) | ||
| ) | ||
| .first() | ||
| ) | ||
|
|
||
| if case and not case.case_type: | ||
| case.case_type = case_type[:200] | ||
| case.updated_at = now |
There was a problem hiding this comment.
Since the backfill logic has been integrated directly into the main transaction block in parse_case_detail, this helper method is no longer needed and can be removed.
Address review feedback on the case_type backfill: - Make revisiting already-enriched blank-case_type rows opt-in via `-a backfill_case_type=true`, so nightly runs never re-fetch rows whose detail page genuinely has no case_type in an endless loop. The historical backfill is run as a one-off Job. - Fold the backfill update into the existing status-check transaction and drop the redundant _backfill_case_type helper/second query. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Addressed the review feedback:
|
1 participant
Summary
~20,368 district-court rows have a blank
case_type. Root cause: districtcase_typeis only ever sourced from the daily cause-list (pesi/daily) listing'scells[3], which is the "मुद्दा विषय" (subject) column — and that cell is blank for many administrative filings. Unlike high/supreme courts, district enrichment never backfillscase_type, so those blanks are permanent. (For reference, high/supreme enrichment readsमुद्दाको किसिमfrom the detail page, which is why they have 0 nulls.)Confirmed against the live detail page that the true case type is available there. Example — case
082-07-0160(kaskidc), currently blank in DB:So the value is recoverable; the district enricher just wasn't reading the right field (it read
मुद्दाको बिषयinto a non-existentcase_subjectattribute, which is silently dropped — noted but out of scope here).Changes
मुद्दाको किसिम→case_typeduring district enrichment, so future cases populate it (mirrors high/supreme).status='enriched'rows whosecase_typeis blank._backfill_case_typepath that only setscase_typeand does not rebuild entities/hearings (the existing rebuild deletesCaseEntityrows that may have downstream linkages).Notes / follow-ups
case_typerows are overwhelmingly non-litigation administrative registrations (marriage, power-of-attorney, etc.); their "type" tends to restate the act. Backfill still makes the data consistent.मुद्दाको बिषय → case_subjectmapping writes to a column that doesn't exist onCourtCase(silent data loss) — worth a separate fix.Test plan
python -m py_compile+ruff checkpassमुद्दाको किसिमfor a currently-blank casescrapy crawl district_case_enrichmentagainst staging; confirm blankcase_typerows get populated and entity rows are untouched🤖 Generated with Claude Code