Update web crawl conditions - #501
Conversation
ppinchuk
added
enhancement
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.
Updates the jurisdiction website discovery / crawl gating logic so that user-supplied jurisdiction websites are treated as ground truth, while website discovery via search happens only when LLM configs are available (and is validated), otherwise website crawling is skipped.
Changes:
- Removed the
validate/validate_user_website_inputswitches and simplified call sites accordingly. - Updated collection steps to (1) normalize a provided website to base domain and (2) only attempt website discovery when LLM models are configured.
- Updated tests and request model-configuration documentation to match the new behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/python/unit/scripts/test_download.py | Updates unit test to match the new find_jurisdiction_website signature. |
| support/jurisdictions/update_jur_websites.py | Updates internal tooling to call find_jurisdiction_website without the removed validate kwarg. |
| compass/scripts/download.py | Removes validate parameter and unconditional “return first candidate” path; website candidates are now always LLM-validated. |
| compass/pipeline/jurisdiction.py | Removes validate_user_website_input from the workflow configuration. |
| compass/pipeline/data_classes.py | Makes model registry available whenever user_model_input is provided; expands model config docs. |
| compass/pipeline/coordinator.py | Removes validate_user_website_input plumbing when creating workflows. |
| compass/pipeline/collection/steps.py | Reworks website resolution: normalize provided URL; only search for a website when LLM models exist; removes user-website validation logic. |
Comments suppressed due to low confidence (1)
compass/scripts/download.py:249
find_jurisdiction_websitewill raise aKeyErrorifmodel_configsis empty or missingLLMTasks.DEFAULT. Since the function is a public helper (also used by tooling), please add explicit input validation with a clear error (e.g., raiseCOMPASSValueErrorstating that a default model config is required for website validation) or handle the no-model case by returningNonebefore attempting validation.
model_config = model_configs.get(
LLMTasks.JURISDICTION_MAIN_WEBSITE_VALIDATION,
model_configs[LLMTasks.DEFAULT],
)
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (57.07%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #501 +/- ##
==========================================
+ Coverage 62.59% 63.15% +0.55%
==========================================
Files 77 77
Lines 7047 7064 +17
Branches 706 703 -3
==========================================
+ Hits 4411 4461 +50
+ Misses 2502 2469 -33
Partials 134 134
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Specifically, we now assume that if a user provides a website via the jurisdictions input, that it is ground truth and does not need to be validated. If they do NOT provide a jurisdiction website but there is a model config we can use, then a search engine search will be applied (along with a validation). If a website is not given and LLM parameters are not available, a web crawl will not happen.