feat: integrate Arctic RL backend (opt-in via [arctic])

[sfc-gh-kganesan]

Opt-in Arctic RL backend for PRIME-RL.

+66 / -2 lines in src/prime_rl/ + packages/prime-rl-configs/ (generic, no integration name)
all integration code under integrations/arctic-rl/  (new directory, ~2.35k LOC)

grep -ri "arctic" src/ packages/ pyproject.toml returns 2 matches, both in docstrings/comments illustrating the new generic field — no imports, no hardcoded references.

Enabling

Any existing recipe flips to Arctic via a one-line overlay — no edits to the base recipe:

uv run rl @ configs/gsm8k/rl.toml @ integrations/arctic-rl/examples/arctic_overlay.toml

The overlay file is 8 lines:

[trainer]
backend = "arctic_rl"

[arctic]
backend = "local"
training_gpus = 1
sampling_tensor_parallel_size = 1
log_prob_gpus = 0

The launcher peeks at trainer.backend before strict config parse and dispatches via importlib.import_module(f"{backend}.entrypoint").main(). Default is "native" — when unset the arctic_rl package is never imported. The base recipe's [inference] block is silently ignored in Arctic mode (Arctic owns sampling via its own server).

Architecture

                       ╔═════════════════════════════════════╗
                       ║  uv run rl @ <recipe>.toml          ║
                       ║                                     ║
                       ║  if trainer.backend != "native":    ║
                       ║    import_module(f"{backend}.       ║
                       ║      entrypoint").main()            ║
                       ║  else: rl(cli(RLConfig))            ║
                       ╚═══════════════╤═════════════════════╝
                                       │ trainer.backend = "arctic_rl"
                                       ▼
                       ┌─────────────────────────────────────┐
                       │  arctic_rl.entrypoint:main          │
                       │  • parse ArcticRLConfig             │
                       │  • rewrite                          │
                       │    orchestrator.student.client      │
                       │    → client_type="custom"           │
                       │    → class_path="arctic_rl          │
                       │      .verifiers_backend             │
                       │      .ArcticClient"                 │
                       └─┬─────────────────────────┬─────────┘
                         │ spawns                  │ spawns
                         ▼                         ▼
   ┌──────────────────────────┐   ┌──────────────────────────────────┐
   │   arctic-trainer         │   │   Orchestrator                   │
   │   (CPU, 1 process)       │   │   (CPU, unchanged)               │
   │                          │   │                                  │
   │   HTTP client to         │   │   resolve_client(...) →          │
   │   Arctic RL:             │   │   ┌──────────────────────────┐   │
   │   /fwd-bwd /step         │   │   │ ArcticClient             │   │
   │   /sync-weights          │   │   │  (verifiers.Client       │   │
   │                          │   │   │   subclass; in-process)  │   │
   └─────┬─────────┬──────────┘   │   │  + 64→1 batcher          │   │
         │         │              │   └────────────┬─────────────┘   │
         │         │              └────────────────┼─────────────────┘
         │  filesystem rollouts +                  │
         │  STABLE marker (unchanged)              │
         ▼                                         ▼
   ╔════════════════════════════════════════════════════════════════╗
   ║                  Arctic RL Server  (GPU)                       ║
   ║  Training Zone  ◄──►  Sampling Zone        Ref Model Zone      ║
   ║   (DeepSpeed/         (vLLM/                (DeepSpeed)        ║
   ║    FSDP/Megatron)      ArcticInference)                        ║
   ║                                                                ║
   ║  Weight Sync — intra-node NCCL between zones                   ║
   ╚════════════════════════════════════════════════════════════════╝

	Native	Arctic
Trainer	torchrun + FSDP2	single CPU process, HTTP client
Inference	vLLM server	(none — ArcticClient runs in-process inside the orchestrator)
Orchestrator	verifiers + rollouts	unchanged; resolves a different Client

Two PRs

1. verifiers — feat(clients): add 'custom' client_type for external Client subclasses sfc-gh-kganesan/verifiers#1 (~30 LOC). Adds client_type="custom" and a class_path: str | None field on ClientConfig. Generic — works for any external Client subclass.
2. prime-rl (this PR). All integration code under integrations/arctic-rl/. ArcticClient(verifiers.OpenAIChatCompletionsTokenClient) overrides three methods: tokenize, get_native_response, from_native_response. No subprocess, no monkey-patch.

Validation

Native-dispatch regression check (4×H200, wandb offline)

Verifies that adding the _peek_backend dispatch + new client_type/class_path config fields doesn't regress the native path. All three runs use trainer.backend = "native" (the default).

Recipe	Model	Steps	Reward (start → end)
examples/reverse_text/rl.toml	Qwen3-0.6B-Reverse-Text-SFT	20/20	0.112 → 0.751
examples/alphabet_sort/rl.toml	Qwen3-4B-Instruct (multi-turn)	10/10	0.472 → 0.435 (200-step recipe truncated)
configs/gsm8k/rl.toml	Qwen3-0.6B (math)	10/10	0.623 → 0.781

Arctic-mode E2E convergence (4×H200, backend = "local", wandb offline)

Two recipes, 30 steps each, end-to-end through the Arctic RL backend.

reverse-text (dedicated example integrations/arctic-rl/examples/arctic_reverse_text/rl.toml)

Step	0	5	10	15	20	25	29
Reward	0.128	0.401	0.627	0.729	0.786	0.781	0.786

0.128 → 0.786 (6.1×), monotonic, plateau by step ~17.

gsm8k (native configs/gsm8k/rl.toml + arctic_overlay.toml, no recipe edits)

Step	0	5	9	11	24	27	29
Reward	0.561	0.670	0.792	0.732	0.754	0.643	0.648

Qwen3-0.6B starts strong on gsm8k; Arctic mode reaches 0.792 peak at step 9, comparable to the native 10-step run (0.781). Late-step variance (~0.55-0.75) is normal RL noise.

Aggregate server-side counters across both runs: 840× HTTP 200, 0× 5xx, ~7,700 rollouts with 0 failures. NCCL weight transfer between training and sampling GPUs averaged 0.9-3.7s (longer for the 2048-token gsm8k responses).

Bugs surfaced + fixed by E2E runs

Six integration bugs uncovered while running real Arctic-mode jobs are folded into the squashed commit:

1. entrypoint.py blanked CUDA_VISIBLE_DEVICES for the trainer subprocess. Correct for backend="remote" (trainer is just an HTTP client), but with backend="local" the trainer spawns the Arctic RL coordinator in-process and Ray needs visible GPUs. Now gated on backend.
2. Ray 2.40+ auto-detects uv-managed venvs (via pyvenv.cfg markers) and re-launches its raylet workers via uv run python, which lands them in a freshly-created empty venv without ray. Stripped uv-related env vars and set RAY_ENABLE_UV_RUN_RUNTIME_ENV=0.
3. The orchestrator's pre-flight wait_for_ready does an HTTP probe on client.base_url. With Arctic mode that URL pointed at the OAI-compat default (localhost:8000), which has nothing listening. Now redirected to arctic.url and skip_model_check=True (the Arctic RL coordinator doesn't speak the OAI catalog API).
4. StaticInferencePool.update_weights issues /pause on its admin clients before pushing weights — fine for vLLM, but the Arctic RL coordinator has no /pause endpoint (it owns its own NCCL weight-sync). Added a no-op override when client_type="custom".
5. ArcticRLConfig validator now silently drops [inference] instead of rejecting it, so existing recipes work via overlay without edits.
6. ArcticTrainerAdapter inspects client.step()'s signature before passing learning_rate (older ArcticRLClient.step() revs take no kwargs).

Out of scope

Multi-node, LoRA, multi-run, teacher_inference, multimodal. Recipes using any of those need explicit edits, the overlay alone won't help.