Rmp plugin - #445
Merged
Merged
Conversation
Add support for extracting geothermal restrictions from BLM RMPs: introduce a large rmp_jurisdictions.csv dataset and register it in the jurisdictions registry. Add compass/utilities/finalize_rmp.py to compile parsed ordinance CSVs, save run metadata, format outputs (qualitative/quantitative), apply empirical adjustments, and generate a run summary. Update one-shot components to include geothermal-specific output columns (location, restriction_type, geothermal_applicability, ammendment). Include an rmp_demo example with configuration, schema, local docs template, plugin config, and a small jurisdictions.csv to demonstrate running the one-shot extraction workflow.
Allow schema-based one-shot plugins to write extracted data via a configurable finalize module (default or "rmp"). Fix PDF reading/OCR: strip unexpected kwargs before calling pdftotext, forward image/convert kwargs into OCR path, and patch pytesseract cleanup on Windows to suppress OSErrors when removing temp files. Adjust finalize_rmp import to match renamed helper, and add local_activate.{sh,bat} plus register the Windows activation script in pixi.toml to support loading a local .env.
* Adjust cleanup function * Minor formatting * Better description * More precise message * Add docs * update local docs * Minor plugin update * Add default dir * Update dep * Remove unused module * Plugins now specify their own output columns * Add extra columns to rmp output * Add `load_dotenv` call * Fix CSV * Use out cols for parser * Fix command call * Fix tests * Wire-up plugin * Update tech * Rename file * Minor bug fix * Update README * Doc fix * Extra columns taken from schema instead
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (71.66%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #445 +/- ##
==========================================
+ Coverage 63.15% 63.25% +0.10%
==========================================
Files 77 78 +1
Lines 7064 7141 +77
Branches 703 710 +7
==========================================
+ Hits 4461 4517 +56
- Misses 2469 2492 +23
+ Partials 134 132 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Bump RMP schema to 1.2.0 and extend extraction schema with new fields (restriction_level, rights_status, binding_status, map_reference) and stricter required properties; clarify handling of "fluid minerals" and broaden guidance for energy/mineral closures. Update plugin_config.yaml to add heuristic keywords (energy, map, mineral/energy terms) and an explicit extraction note about treating fluid minerals as covering geothermal. Extend compass.extraction.rmp __init__.py to expose restriction_level and document_name output columns and override save_structured_data to populate document_name from the source filename. Add debug.log. Note: compass/data/rmp_jurisdictions.csv contains unresolved merge conflict markers (<<<<<<< / >>>>>>>) and needs manual resolution before use.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends COMPASS’ extraction ecosystem by adding a geothermal RMP one-shot example/plugin and by generalizing “finalize” output handling via plugin-defined output columns and optional post-processing, alongside several OCR-related robustness fixes.
Changes:
- Introduces plugin-defined
OUTPUT_COLUMNSand threads them through final DB compilation (doc_infos_to_db/save_db) and CLI finalize flow. - Adds one-shot post-processing support plus a new geothermal RMP extractor (schema + plugin config) and an end-to-end example folder.
- Improves OCR/file-loading behavior (kwargs handling, forwarding OCR kwargs, and Windows-specific pytesseract cleanup patch).
Reviewed changes
Copilot reviewed 30 out of 34 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/python/unit/utilities/test_utilities_finalize.py | Updates unit tests for new doc_infos_to_db / save_db signatures and output-column routing. |
| tests/python/unit/plugin/test_plugin_ordinances.py | Adds required OUTPUT_COLUMNS attribute to test plugins to satisfy updated interface. |
| tests/python/unit/plugin/test_plugin_one_shot_base.py | Adds unit test coverage for deriving output columns from one-shot schema config. |
| tests/python/unit/pipeline/test_pipeline_collection_persistence.py | Adds unit test ensuring docs without parsed text are omitted from persistence output. |
| pyproject.toml | Adds python-dotenv runtime dependency. |
| pixi.toml | Updates python-dotenv pin to match pyproject.toml. |
| pixi.lock | Locks python-dotenv addition. |
| examples/rmp_demo/README.md | Documents how to run the new RMP one-shot example. |
| examples/rmp_demo/one-shot/local_docs.json5 | Provides template for local RMP PDF inputs. |
| examples/rmp_demo/one-shot/config.json5 | Provides example run configuration for the RMP one-shot workflow. |
| examples/rmp_demo/jurisdictions.csv | Adds example jurisdiction input for the demo. |
| compass/utilities/jurisdictions.py | Registers the new rmp_jurisdictions.csv dataset as a known jurisdiction source. |
| compass/utilities/finalize.py | Generalizes final DB creation and CSV writing to use plugin-provided output columns. |
| compass/services/cpu.py | Improves OCR kwargs handling and adds Windows-specific pytesseract cleanup override. |
| compass/scripts/download.py | Tweaks progress description text for document filtering. |
| compass/plugin/post_processing.py | Adds post-processing registry and add_document_name helper. |
| compass/plugin/one_shot/components.py | Refactors default prompts, adds parser “possible output cols” plumbing, and uses instance prompt fields. |
| compass/plugin/one_shot/base.py | Adds output-column derivation from schema and supports configuring post-processing steps. |
| compass/plugin/interface.py | Adds OutputColumn, post-processing support, and makes OUTPUT_COLUMNS part of the plugin contract. |
| compass/plugin/init.py | Re-exports OutputColumn. |
| compass/extraction/wind/plugin.py | Defines OUTPUT_COLUMNS for wind extractor. |
| compass/extraction/solar/plugin.py | Defines OUTPUT_COLUMNS for solar extractor. |
| compass/extraction/small_wind/plugin.py | Defines OUTPUT_COLUMNS for small wind extractor. |
| compass/extraction/rmp/rmp_schema.json | Adds geothermal RMP extraction schema. |
| compass/extraction/rmp/plugin_config.yaml | Adds one-shot plugin config for geothermal RMP extraction (including post-processing steps). |
| compass/extraction/rmp/init.py | Registers the geothermal RMP extractor and customizes the text-collection system prompt. |
| compass/extraction/geothermal_electricity/plugin_config.yaml | Minor formatting-only change. |
| compass/extraction/geothermal_electricity/init.py | Adds geothermal electricity extractor registration via one-shot plugin factory. |
| compass/extraction/init.py | Imports/registers new extractors (geothermal electricity + RMP). |
| compass/data/rmp_jurisdictions.csv | Adds jurisdiction inventory for RMP-style “jurisdictions”. |
| compass/_cli/finalize.py | Updates finalize CLI to compile DB using plugin-specific output columns. |
| compass/init.py | Loads dotenv on import and imports/registers new extractors. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding an example folder for RMPs. This will house the RMP schema as well as the configs that I ran when extracting information related to geothermal restrictions.
Also fixed a problem where I was not able to run a document but the solution does not seem super clean. @ppinchuk would love some ideas for a better way to solve the problem. The five problems fixed during this were:
Problem Symptom
Bad TESSDATA_PREFIX Tesseract can't load language data, fails immediately
OCR kwargs not stripped from non-OCR path Non-OCR reads crash with unexpected kwarg
OCR kwargs not forwarded to read_pdf_ocr DPI/timeout settings silently ignored
Default DPI too high Per-page timeout exceeded on 165-page scan
WinError 32 not suppressed Pages silently drop OCR results on Windows