Skip to content

fix: alert on mirror service outage instead of silently zeroing scores - #820

Closed
wdeveloper16 wants to merge 2 commits into
entrius:testfrom
wdeveloper16:fix/mirror-outage-alerting
Closed

fix: alert on mirror service outage instead of silently zeroing scores#820
wdeveloper16 wants to merge 2 commits into
entrius:testfrom
wdeveloper16:fix/mirror-outage-alerting

Conversation

@wdeveloper16

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #796. When mirror.gittensor.io is unavailable, fetch failures were either logged at WARNING level (issue discovery) or without scoring-impact context (OSS path), making it impossible to distinguish a service outage from isolated miner failures by watching error logs.

  • Upgrade per-miner issue-discovery fetch failure from WARNINGERROR (consistent with OSS path severity)
  • Improve OSS per-miner error message to include hotkey prefix and explicit consequence ("scores will be 0 this round for this miner")
  • Add aggregate outage detection in run_mirror_issue_discovery: emits ERROR when ≥50% of eligible miners fail and total_eligible >= 5 (the guard suppresses false positives in small/dev environments)
  • Add 4 tests covering: total failure fires alert, 50% boundary fires, 20% rate does not fire, small-population guard suppresses

No behaviour changes

Scoring logic is untouched — this is purely observability. All 51 existing tests continue to pass.

Test plan

  • uv run --with pytest pytest tests/validator/oss_contributions/mirror/test_load.py tests/validator/issue_discovery/test_mirror_scan.py -v → 51 passed

@xiao-xiao-mao xiao-xiao-mao Bot added the enhancement New feature or request label Apr 27, 2026
@wdeveloper16
wdeveloper16 force-pushed the fix/mirror-outage-alerting branch 9 times, most recently from a4a0681 to 2486936 Compare April 29, 2026 15:54
@wdeveloper16
wdeveloper16 force-pushed the fix/mirror-outage-alerting branch from 2486936 to 853427e Compare April 29, 2026 17:52
@anderdc

anderdc commented Apr 29, 2026

Copy link
Copy Markdown
Collaborator

Needless logging change. Closing

@anderdc anderdc closed this Apr 29, 2026
@wdeveloper16
wdeveloper16 deleted the fix/mirror-outage-alerting branch April 29, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants