Skip to content

developer.md

Latest commit

 

History

History
382 lines (270 loc) · 11.9 KB

developer.md

File metadata and controls

382 lines (270 loc) · 11.9 KB
Raw

Developer Guide

This document is for people running or changing the Gradients on Demand repository. It covers the repo layout, the main runtime components, and the common commands for validators, trainers, miners, tests, and local development.

Repository Layout

The repo is organized by runtime boundary:

Path Purpose
core/ Shared contracts, constants, logging, dataset whitelist data, training templates, and narrow helpers used by more than one runtime.
miner/ The public miner API. Miners answer tournament repo requests from validators.
trainer/ The trainer service that runs miner training repos inside Docker, manages GPU availability, caches models/datasets, and uploads finished models.
validator/ Validator API, tournament lifecycle, task creation, evaluation, scoring, persistence, infrastructure integrations, and transfer accounting.
ops/ Dockerfiles, compose files, config generation, observability, manual probes, tournament tools, and operational scripts.
tests/ Unit and integration tests grouped by domain.
docs/ The two maintained docs: this guide and miner.md.

Inside validator/, the major boundaries are:

Path Purpose
validator/app/ App config and FastAPI dependencies.
validator/db/ Database connection, migrations, constants, and SQL access modules.
validator/endpoints/ Validator API endpoints.
validator/evaluation/ Evaluation runtimes, model checks, PvP evaluation, Docker/Basilica execution, and evaluator entrypoints.
validator/infrastructure/ MinIO, Redis/cache policy, substrate, content service, LLM, retries, and external service wrappers.
validator/lifecycle/ Main validator task lifecycle loops.
validator/nodes/ Metagraph/node refresh logic.
validator/scoring/ Task scoring, tournament scoring, scoring models, constants, and weight setting.
validator/tasks/ Task schemas, task config, task requests, task details, dataset prep, model prep, rewards, and synthetic task creation.
validator/tournament/ Tournament orchestration, participants, rounds, brackets, reports, GPU requirements, GitHub validation, and obfuscation detection.
validator/transfers/ Transfer and balance schemas.

Avoid new utils.py, shared/, or cross-domain catch-all modules. If code belongs to scoring, put it in validator/scoring; if it belongs to task prep, put it in validator/tasks/prep; if both validator and trainer need it, consider core/.

Runtime Topology

There are three services you usually care about:

Service Command Default port Role
Miner task miner 7999 Returns a miner's training repository for each tournament type.
Trainer task trainer 8001 Runs training jobs on GPUs, serves trainer status to the validator.
Validator task validator or task validator_dev 9001 Creates tasks/tournaments, talks to miners and trainers, evaluates submissions, sets weights.

The validator depends on Postgres, Redis, S3-compatible storage, Docker, and access to Bittensor/Fiber. Trainer nodes need Docker with GPU support, Hugging Face credentials, and enough disk for model/dataset/cache volumes.

Setup

Clone the repo:

git clone https://github.com/rayonlabs/G.O.D.git
cd G.O.D

Install system dependencies on a fresh Ubuntu machine:

task bootstrap

Install the Python package:

task install

For development, install dev dependencies and pre-commit:

pip install -e '.[dev]'
pre-commit install

If you need GPU evaluation dependencies locally:

pip install -e '.[dev,gpu]'

Configuration Files

Configuration is generated through ops.tools.config.create_config and written to env files in the repo root.

Config Command Output
Validator task config .vali.env
Validator dev python -m ops.tools.config.create_config --dev .vali.env
Miner task miner-config .1.env
Trainer task trainer-config .trainer.env
Auditor task auditor-config .test-temp.env

Validator config prompts for wallet, hotkey, subtensor network, DB settings, S3-compatible storage, validator port, and whether to set metagraph weights. After generating .vali.env, add a model hash salt:

echo "MODEL_HASH_SALT=$(openssl rand -hex 32)" >> .vali.env

For local/dev validator features that call the content service, add:

echo "NINETEEN_API_KEY=<your-nineteen-api-key>" >> .vali.env

For SWE Infinite environment tournament evaluation, configure the external Affinetes server. The validator config generator prompts for this optional URL; for an existing .vali.env, add:

echo "SWE_INFINITE_SERVER_BASE_URL=<https://your-affinetes-swe-server>" >> .vali.env

The SWE Infinite server should run on infrastructure separate from the validator. It evaluates models by calling the SGLang URL that the evaluator sends in each request, so the SWE server host does not need GPUs, but it does need:

  • Docker Engine installed and the Docker daemon running.
  • The host Docker socket mounted into the SWE server container, because the server pulls and starts per-task Docker images.
  • Enough disk for SWE task checkouts, temporary files, and container layers.
  • Outbound network access to the model server URL, GitHub, package indexes, and any task dependencies the SWE image fetches.
  • An inbound HTTP route from the validator/Basilica eval container to the server port.

Set Docker Hub credentials on the SWE server host before starting the container. The Affinetes SWE server reads these env vars and runs docker login on startup, which avoids anonymous pull rate limits and makes task image pulls faster and more reliable:

export DOCKER_HUB_USERNAME=<dockerhub-username>
export DOCKER_HUB_TOKEN=<dockerhub-access-token>

Run the published SWE Infinite image:

docker pull gradientsio/swe-infinite:v1
docker rm -f swe-infinite || true
docker run -d \
  --restart unless-stopped \
  --name swe-infinite \
  -p 8000:8000 \
  -e DOCKER_HUB_USERNAME \
  -e DOCKER_HUB_TOKEN \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gradientsio/swe-infinite:v1

The --restart unless-stopped policy makes Docker restart the SWE server after host reboots. On systemd hosts, also make sure Docker itself starts on boot:

sudo systemctl enable --now docker
docker update --restart unless-stopped swe-infinite

Then check it locally and expose the same base URL in .vali.env:

curl http://127.0.0.1:8000/health
echo "SWE_INFINITE_SERVER_BASE_URL=http://<swe-server-host>:8000" >> .vali.env

If the server is behind a reverse proxy or load balancer, set SWE_INFINITE_SERVER_BASE_URL to that public HTTPS URL instead. The evaluator uses Affinetes' /call endpoint by default. SWE Infinite evaluator behavior is configured in validator/evaluation/swe_infinite_config.py; production task range and task count stay in core/constants/environments.py. The metadata URL is only used to resolve task_id_max if the SWE EnvironmentConfig.task_id_max sentinel is set to 0.

Trainer config needs at least:

HUGGINGFACE_TOKEN=<token>
HUGGINGFACE_USERNAME=<username>
WANDB_TOKEN=<token>
ORCHESTRATOR_IPS=<validator-ip-or-comma-separated-ips>

ORCHESTRATOR_IPS controls which validator IPs may call the trainer. Localhost is always allowed.

Running A Validator

For a normal validator node:

task config
task install
task validator

task validator starts the compose stack in ops/compose/docker-compose.yml, refreshes Grafana, and launches ops/validator_ops/start_validator.sh.

For a validator with auto-updates managed by PM2:

task autoupdates

For local development:

task config
task validator_dev

task validator_dev starts the base compose stack plus ops/compose/docker-compose.dev.yml, applies DB migrations through dbmate, and starts the validator.

Useful DB commands:

task dbup
task dbdown
task postgres
task db-dump
task db-restore

Running A Trainer On A New Machine

Trainer nodes execute miner code. They need Docker, NVIDIA drivers, NVIDIA Container Toolkit, the repo, and the Python package installed.

  1. Clone and install:
git clone https://github.com/rayonlabs/G.O.D.git
cd G.O.D
task bootstrap
task install
  1. Generate trainer config:
task trainer-config
  1. Edit .trainer.env if needed:
ORCHESTRATOR_IPS=<validator-ip>
HUGGINGFACE_TOKEN=<token>
HUGGINGFACE_USERNAME=<username>
WANDB_TOKEN=<token>
  1. Start the trainer:
task trainer
  1. Verify GPU visibility:
curl http://localhost:8001/v1/trainer/get_gpu_availability

The trainer builds miner-provided Dockerfiles, starts training containers, streams logs, writes model outputs under the checkpoints Docker volume, caches models/datasets under the cache Docker volume, and uploads finished models to Hugging Face.

Trainer API endpoints are defined in validator/infrastructure/service_constants.py and registered in trainer/endpoints.py.

Running A Miner Locally

Generate miner config:

task miner-config

Start the miner:

task miner

Check the tournament repo endpoint:

curl http://localhost:7999/training_repo/text

The endpoint implementation is miner/endpoints/training_repo.py. Tournament miners should read docs/miner.md.

Observability

The validator can host Grafana/Loki/Prometheus for trainer logs.

On the validator:

task deploy-observability-server

Optional .vali.env overrides:

OBSERVABILITY_DOMAIN=<domain-or-ip>
GRAFANA_TRAINING_PASSWORD=<password>
LOKI_PASSWORD=<password>
GRAFANA_ANONYMOUS_ENABLED=false

On each trainer node:

task deploy-trainer-logs

The trainer log shipper requires these values in .trainer.env:

LOKI_ENDPOINT=https://<validator-host>:3101
LOKI_PASSWORD=<same-password-as-validator>
LOKI_USERNAME=trainer

Useful commands:

task logs-observability
task logs-trainer-shipper
task status-trainer-logs
task test-trainer-logs

Trainer log shipping uses Vector and collects containers named like text-trainer-*, image-trainer-*, downloader-*, and hf-upload-*.

Evaluations And Manual Tools

Run the evaluation helper:

python -m ops.validator_ops.run_evaluation --help

Re-evaluate an existing task:

python -m ops.validator_ops.run_evaluation --task_id <task_id>

Evaluate a non-submitted model against a task:

python -m ops.validator_ops.run_evaluation --task_id <task_id> --models <model_name>

Tournament and debugging tools live under:

ops/tools/tournament/
ops/tools/evaluation/
ops/tools/scoring/
ops/tools/simulations/
ops/examples/
ops/runbooks/

Testing And Linting

Run lint:

uv run --extra dev ruff check core miner trainer validator ops tests

Run tests:

uv run --extra dev pytest -q

Run tests that need GPU/evaluation dependencies:

uv run --extra dev --extra gpu pytest -q

Run a focused test:

uv run --extra dev pytest -q -o addopts='' tests/validator/scoring/test_tournament_scoring_pipeline.py

pytest defaults to random ordering through pyproject.toml; use -o addopts='' when you need deterministic focused debugging.

Development Notes

  • Keep imports pointing at the owner module. Do not rely on compatibility imports from unrelated domains.
  • Keep constants near the thing that owns them. Docker/service endpoints live under infrastructure constants, scoring constants under scoring, tournament shape under tournament, task prep constants under task prep.
  • Keep miner-facing contracts in core/models/payload_models.py and the focused owner modules under core/models/ when they cross process boundaries.
  • Keep validator-only schemas in the validator domain that owns them.
  • Prefer focused tests around changed behavior. For pure reorg work, run import smoke, compileall, and undefined-name lint.