Skip to content
11 changes: 6 additions & 5 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ those documents into a versioned database that downstream users can query as a C
workbook, or GeoPackage.

.. image:: https://raw.githubusercontent.com/NatLabRockies/COMPASS/main/docs/source/_static/overview.png
:alt: High-level overview of the INFRA-COMPASS pipeline
:alt: High-level overview of the COMPASS pipeline
Comment thread
castelao marked this conversation as resolved.
:align: center
:width: 100%

Expand All @@ -64,17 +64,18 @@ architecture around the LLM call:
jurisdictions.


Read more about the tool in the `documentation <https://natlabrockies.github.io/COMPASS/misc/about.html>`_.
Read more about the tool in the `documentation <https://natlabrockies.github.io/COMPASS/misc/about.html>`_,
or check out summaries of the validation results in the `validation report <https://natlabrockies.github.io/COMPASS/val/validation.html>`_.


Where is the extracted ordinance data?
======================================
The National Laboratories of the Rockies (NLR) typically runs the INFRA-COMPASS pipeline
The National Laboratories of the Rockies (NLR) typically runs the COMPASS pipeline
annually and publishes refreshed datasets to OpenEI. The latest published ordinance datasets
are available here:

- Solar: https://data.openei.org/submissions/8519
- Wind: https://data.openei.org/submissions/8602
- Solar: https://data.openei.org/submissions/8602
- Wind: https://data.openei.org/submissions/8519
Comment thread
ppinchuk marked this conversation as resolved.


Installing COMPASS
Expand Down
4 changes: 4 additions & 0 deletions docs/source/misc/about.rst
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,7 @@ See the development guides for full details:
`plugin development <https://natlabrockies.github.io/COMPASS/dev/plugin_development.html>`_
and
`advanced plugin development <https://natlabrockies.github.io/COMPASS/dev/advanced_plugin_development.html>`_.

Validation
==========
Take a look at some high-level validation results in the `validation report <https://natlabrockies.github.io/COMPASS/val/validation.html>`_.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/val/solar_val_aug_2025.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
91 changes: 91 additions & 0 deletions docs/source/val/validation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,95 @@ Here we give a brief overview of the results of known COMPASS validation efforts

COMPASS validation is an ongoing effort, and we will update this page as new results become available.

Geothermal Electricity Document Collection Validation (May 2026)
----------------------------------------------------------------
This validation was for the document collection portion only.

Info
^^^^

- **COMPASS Version**: `v0.15.2 <https://github.com/NatLabRockies/COMPASS/releases/tag/v0.15.2>`_
- **Number of Documents**: 100 (Assuming 10,000 jurisdictions, there is a 95% chance that the metrics are within ±9.75% of the reported value)
- **Features**: None
- **Procedure Validated**: Document collection from web
- **LLM(s) used**: OpenAI GPT-4.1, OpenAI GPT-5


Results
^^^^^^^

.. image:: geothermal_electricity_web_scraping_may_2026.png


Wind Document Collection Validation (September 2026)
Comment thread
ppinchuk marked this conversation as resolved.
Outdated
----------------------------------------------------
This validation was for the document collection portion only.

Info
^^^^

- **COMPASS Version**: `v0.8.2 <https://github.com/NatLabRockies/COMPASS/releases/tag/v0.8.2>`_
- **Number of Documents**: 100 (Assuming 10,000 jurisdictions, there is a 95% chance that the metrics are within ±9.75% of the reported value)
- **Features**: None
- **Procedure Validated**: Document collection from web
- **LLM(s) used**: OpenAI GPT-4o-mini, OpenAI GPT-4.1-nano, OpenAI GPT-4.1-mini, OpenAI GPT-4.1


Results
^^^^^^^

.. image:: wind_web_scraping_september_2025.png


Solar Validation (August 2025)
------------------------------

This validation was for the ordinance value extraction portion only (assume documents are correct and belong
to the correct jurisdiction).

This validation focused on the model ability to extract structured ordinance data from unstructured wind ordinance text documents.


Info
^^^^

- **COMPASS Version**: `v0.7.0 <https://github.com/NatLabRockies/COMPASS/releases/tag/v0.7.0>`_
- **Number of Documents**: 78 (Assuming 10,000 jurisdictions, there is a 95% chance that the metrics are within ±11.05% of the reported value)
- **Features**:

- structures (participating)
- property line (participating)
- structures (non-participating)
- property line (non-participating)
- roads
- railroads
- transmission
- water
- noise
- maximum height
- maximum project size
- minimum lot size
- maximum lot size
- density
- coverage
- prohibitions
- decommissioning
- glare
- visual impact
- primary use districts
- special use districts
- accessory use districts
- ordinance effective year


- **Procedure Validated**: Ordinance extraction from documents
- **LLM(s) used**: OpenAI GPT-4.1-nano, OpenAI GPT-4.1-mini, OpenAI GPT-4.1

Results
^^^^^^^

.. image:: solar_val_aug_2025.png


Mini Wind Validation (June 2025)
--------------------------------
Expand Down Expand Up @@ -104,3 +193,5 @@ Results
^^^^^^^

.. image:: wind_jan_2024.png

.. Margin of error calculator: https://www.calculator.net/sample-size-calculator.html
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
326 changes: 326 additions & 0 deletions support/validation/May_2026_Geothermal_webscraping_validation.ipynb

Large diffs are not rendered by default.

File renamed without changes.