Skip to content

fix: use SOURCE-quality score for eligibility gates - #1024

Closed
bitloi wants to merge 6 commits into
entrius:testfrom
bitloi:fix/1022-source-quality-gates
Closed

fix: use SOURCE-quality score for eligibility gates#1024
bitloi wants to merge 6 commits into
entrius:testfrom
bitloi:fix/1022-source-quality-gates

Conversation

@bitloi

@bitloi bitloi commented May 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Eligibility-style gates were using aggregate token_score, which includes tree-diff TEST score, even though the base-score threshold already uses SOURCE-only quality. A large test-only PR could therefore get only the intended small contribution bonus, but still count as a valid merged PR, valid solved issue-discovery PR, or pioneer-eligible PR.

This change:

  • Persists source_token_score from calculate_base_score_for_pr_files().
  • Keeps aggregate token_score unchanged for contribution accounting and token totals.
  • Uses SOURCE-quality score for merged PR eligibility, mirror valid-solved gates, and legacy/mirror pioneer eligibility.
  • Falls back to aggregate token_score when source_token_score is absent so older/test objects remain compatible.
  • Adds regression coverage showing TEST-only PRs can earn the small bonus without satisfying SOURCE-quality gates.

Related Issues

Fixes #1022.

Related context:

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Documentation
  • Other (describe below)

Testing

  • Tests added/updated
  • Manually tested

Commands run:

git diff --check
uv run pytest tests/validator/test_source_quality_gates.py tests/validator/issue_discovery/test_mirror_scan.py tests/validator/oss_contributions/mirror/test_base_score_helper.py tests/validator/oss_contributions/mirror/test_scored_pr.py tests/validator/test_pioneer_dividend.py -q
uv run ruff check gittensor/classes.py gittensor/constants.py gittensor/validator/issue_discovery/mirror_scan.py gittensor/validator/issue_discovery/scoring.py gittensor/validator/oss_contributions/credibility.py gittensor/validator/oss_contributions/mirror/adapters.py gittensor/validator/oss_contributions/mirror/scored_pr.py gittensor/validator/oss_contributions/mirror/scoring.py gittensor/validator/oss_contributions/scoring.py tests/validator/issue_discovery/test_mirror_scan.py tests/validator/oss_contributions/mirror/test_base_score_helper.py tests/validator/oss_contributions/mirror/test_scored_pr.py tests/validator/test_pioneer_dividend.py tests/validator/test_source_quality_gates.py
uv run ruff format --check gittensor/classes.py gittensor/constants.py gittensor/validator/issue_discovery/mirror_scan.py gittensor/validator/issue_discovery/scoring.py gittensor/validator/oss_contributions/credibility.py gittensor/validator/oss_contributions/mirror/adapters.py gittensor/validator/oss_contributions/mirror/scored_pr.py gittensor/validator/oss_contributions/mirror/scoring.py gittensor/validator/oss_contributions/scoring.py tests/validator/issue_discovery/test_mirror_scan.py tests/validator/oss_contributions/mirror/test_base_score_helper.py tests/validator/oss_contributions/mirror/test_scored_pr.py tests/validator/test_pioneer_dividend.py tests/validator/test_source_quality_gates.py
uv run pyright gittensor/classes.py gittensor/validator/issue_discovery/mirror_scan.py gittensor/validator/oss_contributions/credibility.py gittensor/validator/oss_contributions/mirror/scored_pr.py gittensor/validator/oss_contributions/mirror/scoring.py gittensor/validator/oss_contributions/scoring.py tests/validator/test_source_quality_gates.py
uv run pytest tests/ -q

Results:

  • Focused regression suite: 68 passed
  • Full test suite: 769 passed, 2 warnings
  • Ruff check: passed
  • Ruff format check: passed
  • Pyright on touched files: 0 errors, 0 warnings

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Changes are documented (if applicable)

@bitloi
bitloi marked this pull request as ready for review May 5, 2026 23:01
@xiao-xiao-mao xiao-xiao-mao Bot added the bug Something isn't working label May 5, 2026
@anderdc

anderdc commented May 10, 2026

Copy link
Copy Markdown
Collaborator

Maintainer review already rejects test-only PRs, so reaching the failure mode this defends against — five test-only PRs accumulating aggregate token_score ≥ 5 from the 0.05× TEST weight — requires maintainer cooperation that doesn't happen in practice. Not pursuing the fix. Closing.

@anderdc anderdc closed this May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Eligibility gates use aggregate token_score instead of SOURCE-quality score

2 participants