Add complexipy

[ppinchuk]

Add complexipy as a linter in CI

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.12587% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.35%. Comparing base (1ba6355) to head (9ffae98).

Files with missing lines	Patch %	Lines
compass/pipeline/data_classes.py	17.64%	14 Missing ⚠️
compass/llm/config.py	50.00%	7 Missing ⚠️
compass/web/file_loader.py	76.00%	4 Missing and 2 partials ⚠️
compass/validation/content.py	73.68%	2 Missing and 3 partials ⚠️
compass/utilities/parsing.py	70.00%	2 Missing and 1 partial ⚠️
compass/common/base.py	0.00%	1 Missing ⚠️
compass/web/search.py	90.00%	0 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (74.12%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #530      +/-   ##
==========================================
+ Coverage   63.95%   64.35%   +0.40%     
==========================================
  Files          78       78              
  Lines        7446     7491      +45     
  Branches      755      754       -1     
==========================================
+ Hits         4762     4821      +59     
+ Misses       2524     2512      -12     
+ Partials      160      158       -2     

Flag	Coverage Δ
unittests	64.35% <74.12%> (+0.40%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds complexipy complexity checking to the Python CI lint job and refactors several Python modules to reduce reported cyclomatic complexity (primarily by extracting helper functions).

Changes:

• Add complexipy to Python dev dependencies and introduce [tool.complexipy] configuration.
• Extend the Python CI lint workflow to run Complexipy alongside Ruff lint/format checks.
• Refactor multiple functions across web, pipeline, validation, and config modules to reduce complexity.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pyproject.toml	Adds complexipy to dev extras and configures [tool.complexipy].
pixi.toml	Adds complexipy to the python-dev feature PyPI dependencies.
pixi.lock	Locks complexipy wheels across platforms/environments.
.github/workflows/ci-python.yml	Adds a Complexipy step to the lint job.
compass/web/website_crawl.py	Extracts helper functions and simplifies link sanitization/blacklist checks.
compass/web/search.py	Extracts helpers for parsing lists and blacklist/whitelist matching.
compass/web/file_loader.py	Extracts helper functions for failed-search collection and ELM/doc selection.
compass/validation/content.py	Refactors chunk-selection logic into async helpers.
compass/utilities/parsing.py	Extracts PDF raw-page handling into a helper.
compass/services/cpu.py	Adds Complexipy ignore directive for a helper.
compass/plugin/one_shot/base.py	Refactors heuristic keyword normalization into smaller helpers.
compass/pipeline/data_classes.py	Extracts helper functions for task/config building and validation.
compass/pipeline/collection/persistence.py	Extracts manifest loading and duplicate-jurisdiction validation.
compass/pipeline/collection/base.py	Refactors execution flow and persists step state on the collector instance.
compass/llm/config.py	Adds safe token-count fallback and refactors client kwarg env fallback logic.
compass/common/base.py	Moves Ruff ignore directive to align with updated function structure.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[castelao]

Indeed it is interesting how it forces the code to be more readable. I agree that it is a good addition, knowing that in some rare moments we might want to bypass it, as you already did in this PR. Thanks for adding it!