Insufficient test coverage and CI coverage for production validator

[joellidin]

Summary

The Crusades validator infrastructure lacks sufficient automated testing and continuous integration coverage to reliably detect outages, API failures, and evaluation pipeline breaks in production.

Recent Incident

On [current date], the validator API (69.19.136.151:8080) became unresponsive, resulting in:

• Complete halt of submission evaluations
• Miner submissions queueing indefinitely without feedback
• Web dashboard showing stale data (last update ~April 2)
• No automated alerting to operators

The outage was only detected via manual miner reports rather than automated monitoring.

Proposed Requirements

1. Health Check & Monitoring

• Automated health checks for validator API endpoints
• Alerting mechanism (Discord/Slack/PagerDuty) for operator notification
• Dashboard heartbeat endpoint for external monitoring

2. Test Coverage

• Unit tests for evaluation pipeline components
• Integration tests for Docker evaluation environment
• End-to-end tests for full submission flow (commit → reveal → evaluate → score)
• Mock tests for Basilica cloud integration path

3. CI/CD Pipeline

• Automated test execution on PR
• Docker image build verification
• Security scanner validation tests
• Test coverage reporting (target: >80%)

4. Operational Resilience

• Automatic retry logic for transient failures
• Queue depth monitoring with thresholds
• Graceful degradation paths when components fail

Context

As discussed with @distributed_tensor and @adamw — the current validator requires manual human monitoring ("1 human watching it") which is not sustainable. Automated detection of validator outages should trigger operator alerts without relying on manual miner reports.

Priority

High — Production stability impacts all miners on subnet 3.

[machine-god-deus]

Machine God picked this up and is drafting a spec now.

[machine-god-deus]

Machine God approved the implementation spec and has begun implementation.

Approved spec

Spec written to workspace/spec.md.

Summary: Seven-phase plan covering test infrastructure, unit tests (error codes, models, DB CRUD, weight math, config), API integration tests (all 13 endpoints + auth), enhanced health endpoint with staleness detection, a Discord-alerting health monitor script, a CI workflow running pytest+coverage+lint on every PR, and Docker Compose integration for the monitor.

Four parallel work units identified. Key risks flagged: DatabaseClient vs Database interface mismatch, config import side effects, and a coverage ceiling around 60% until the monolithic runner/validator files are refactored.

[machine-god-deus]

Machine God completed delivery.

Pull requests:

• Insufficient test coverage and CI coverage for production validator #81