[integrations] Land verl RemoteBackend adapter under arctic_platform/integrations/verl/

[sfc-gh-kganesan]

Summary

Land the ArcticRL ⇄ verl integration as an arctic_platform/integrations/verl/ subpackage, plugged into verl core via the VERL_USE_EXTERNAL_MODULES hook. This unblocks the currently-stalled verl-project/verl#6422 by moving all Arctic-specific code out of the verl source tree while keeping the generic RemoteBackend abstraction in verl core.

Target UX:

pip install arctic-platform[verl]
export VERL_USE_EXTERNAL_MODULES=arctic_platform.integrations.verl.register
verl train ... trainer.remote_backend=arctic

Three commands. Zero verl fork required. Snowflake owns both the RL runtime (arctic_platform.rl) and the verl adapter in one repo.

Motivation

• PR #6422 is stuck. Approved by @zw0610 on 2026-06-01, then on 2026-06-30 @wuxibin89 requested (a) port to the V1 trainer at verl/trainer/ppo/v1/trainer_base.py, and (b) "keep remote_backend abstraction in verl and separate all arctic_rl implementation to external module. All specific remote backend implementation should be dynamic plugin by hook VERL_USE_EXTERNAL_MODULES".
• Precedent exists for framework-aware code in this repo. arctic_platform/rl/processors/verl_grpo.py (~440 LOC) is already a verl-shaped loss registered via arctic_platform.rl's pipeline. Formalizing this under integrations/verl/ cleans up an implicit habit.
• Precedent exists for optional extras. pyproject.toml already ships [rl], [dev], [testing], [formatting]. Adding [verl] is one block, no new mechanism.
• Owning both sides in one repo removes cross-repo lag. During PR RL correctness ports #3 review, adapter changes were blocked waiting for arctic_training async signatures (commit af1ab8d). Same repo = same PR.

Design

Package layout (added under arctic_platform/integrations/verl/)

arctic_platform/
    integrations/                                       # NEW
        __init__.py
        verl/
            __init__.py
            register.py                                  # ~15 LoC entry hook
            adapter.py                                   # ← was verl/workers/remote_client/arctic_rl.py (565 LoC)
            rollout.py                                   # ← was verl/workers/rollout/remote_rollout/arctic_rollout/ (339 LoC)
            worker.py                                    # ← was verl/remote_backend/workers/arctic_rl/worker.py (217 LoC)
            grpo_loss.py                                 # ← was arctic_platform/rl/processors/verl_grpo.py (~440 LoC)
            config/arctic.yaml                           # ← was verl/trainer/config/remote_backend/arctic.yaml
            examples/
                run_bird_grpo_arl.sh
                run_gsm8k_grpo_arl.sh
            README.md
    rl/
        processors/verl_grpo.py                          # backward-compat shim: re-exports from integrations/verl/grpo_loss.py
pyproject.toml                                            # +5 LoC: [verl] extra

register.py (the single file VERL_USE_EXTERNAL_MODULES imports)

\"\"\"Entry point for verl-side registration of the Arctic RL backend.

Import triggered by:
    export VERL_USE_EXTERNAL_MODULES=arctic_platform.integrations.verl.register
\"\"\"
from verl.remote_backend.base import RemoteBackendRegistry
from verl.workers.rollout.replica import RolloutReplicaRegistry


def _load_arctic_backend():
    from arctic_platform.integrations.verl.adapter import ArcticRLClientWrapper
    return ArcticRLClientWrapper


def _load_arctic_replica():
    from arctic_platform.integrations.verl.rollout import ArcticRolloutReplica
    return ArcticRolloutReplica


RemoteBackendRegistry.register(\"arctic\", _load_arctic_backend)
RolloutReplicaRegistry.register(\"arctic\", _load_arctic_replica)

pyproject.toml addition

[project.optional-dependencies]
verl = [\"hydra-core>=1.3\"]  # verl itself is user-supplied; version pins live in launchers

Backward-compat shim

Any downstream user still doing from arctic_platform.rl.processors.verl_grpo import ... continues to work:

# arctic_platform/rl/processors/verl_grpo.py  (after refactor)
\"\"\"Compat shim: verl-shaped loss now lives under integrations/verl/.\"\"\"
from arctic_platform.integrations.verl.grpo_loss import *  # noqa: F401, F403

Companion work (verl core)

This issue is scoped to arctic-platform. The paired verl-core change lands in verl-project/verl#6422:

• Adds verl/remote_backend/{base,trainer_v1,worker,worker_utils}.py — ABC + registry + V1-hook-based trainer + generic forwarder worker. Zero Arctic-specific code.
• Ports RemoteBackendTrainer from RayPPOTrainer subclass → PPOTrainer V1-hook subclass, overriding _compute_old_log_prob, _compute_ref_log_prob, _update_actor, _save_checkpoint.
• Enforces RemoteBackend.requires_single_forwarder() (Samyam's original constraint).
• Deletes verl/workers/remote_client/arctic_rl.py, verl/workers/rollout/remote_rollout/arctic_rollout/, verl/remote_backend/workers/arctic_rl/, verl/trainer/config/remote_backend/arctic.yaml, examples/arctic_rl/* from the PR diff.

Net verl-core diff: ~600 LoC added, ~1120 LoC removed (net negative — we're taking Arctic code out of verl core).

Deliverables

• PR A — this repo (Arctic-Platform): add arctic_platform/integrations/verl/ package, add [verl] extra, add backward-compat shim for rl/processors/verl_grpo.py
• PR B — [trainer] feat: generic remote backend abstraction for RL Training verl-project/verl#6422: update in place to strip all Arctic code + port trainer to V1 hooks
• E2E golden-run smoke passes on both branches
• Update PR #6422 body with new checklist + reference to this issue
• Update verl-project/verl RFC #6537 with the plugin-hook direction

Timeline

Engineering: 5–7 days. Calendar: ~2 weeks with normal review latency.

Day	Track	Task	Deliverable
1	Design	Read V1 PPOTrainer, TrainingWorker, LLMServerClient end-to-end; DM 1-page design proposal to @wuxibin89 / @zw0610	Design doc
2	A (this repo)	Open PR A with integrations/verl/ scaffolding + backward-compat shim + [verl] extra	PR A ready
2–3	B (verl)	Rewrite remote_backend/trainer.py as V1 hooks against PPOTrainer	Local diff
3–4	B (verl)	Update PR #6422 in place: retarget to main, delete arctic files, add V1 trainer, update main_ppo.py routing	PR #6422 updated
5	E2E	Golden-run smoke: 4-step 0.6B GSM8K + 4-step 0.6B BIRD; compare to reference	Green smoke logs
5–6	CLA	Get all 4 unsigned committers on PR #6422 to sign	CLA green
7–14	Review	Xibin/Zhi review, address feedback, second smoke round	Merge on both sides

Testing strategy

Layered from cheap-static to expensive-E2E. Each layer catches a specific class of bug.

Layer 0 — Static (blocks PR)

• pre-commit passes (ruff, ruff-format, mypy, license, docstrings)
• Type annotations on all new public classes

Layer 1 — Unit tests (verl PR)

• verl/tests/remote_backend/test_registry.py — register/get/create round-trip, duplicate register, lazy load
• verl/tests/remote_backend/test_trainer_asserts.py — _enforce_single_forwarder_if_required() matrix
• verl/tests/remote_backend/test_hooks_lifecycle.py — V1 hook fire order, destroy() on exception paths
• verl/tests/remote_backend/test_worker_utils.py — left_pad_to, make_njt, normalize_backend_metrics
• verl/tests/remote_backend/test_v1_dispatch.py — mesh dispatch with world_size=1
• Ship a MockRemoteBackend fixture in verl/remote_backend/testing.py

Layer 2 — Unit tests (this repo)

• arctic_platform/tests/integrations/verl/test_register.py — VERL_USE_EXTERNAL_MODULES import triggers registration on both RemoteBackendRegistry and RolloutReplicaRegistry; lazy loaders defer heavy imports
• arctic_platform/tests/integrations/verl/test_adapter.py — ArcticRLClientWrapper.from_config(), reconnect_handle(), idempotent destroy()
• arctic_platform/tests/integrations/verl/test_payload.py — golden-file snapshot of _prepare_padded_arctic_batch_dict() wire format (catches wire regressions immediately)
• arctic_platform/tests/integrations/verl/test_backward_compat.py — arctic_platform.rl.processors.verl_grpo import still works

Layer 3 — Cross-repo integration

• test_e2e_mock.py in verl PR — 4-step train loop against MockRemoteBackend, CPU only, ~2 min
• arctic_platform's @pytest.mark.integration_verl job — pins verl PR branch, uses MockArcticServer (in-process stub of ArcticRLRayClient), ~5 min

Layer 4 — E2E smoke (real Arctic backend, GPU)

Reference numbers to match, snapshotted from proven-good runs before any refactor:

• Golden Run 1 — 0.6B GSM8K, 4 steps. Reproduce Jun-16 numbers within ±5% reward, ±10% MFU.
• Golden Run 2 — 0.6B BIRD, 4 steps. Match PR Fix zorro labels bug #6 reference:
  • step 1–4 reward: 0.290 / 0.293 / 0.276 / 0.306 (±5%)
  • val bird/reward: 0.2941 (±5%)
  • val exec: 0.5222 (±5%)
  • val format: 0.9426 (±5%)
• Golden Run 3 — 8B BIRD, 20 steps. Reward monotone-increasing trend, no divergence, no OOM. Compare to arctic_v5 trajectory (~4h on 1 node / 8 H200).
• Golden Run 4 (optional) — 32B BIRD, 4-node, 5 steps. Match AUTONOMOUS_STATUS.md steady-state timings within ±15%.

Merge safety guardrails

• G1. Feature-flag default. trainer.remote_backend: null in ppo_trainer.yaml → exact same code path as today. Existing verl users cannot regress. Verified by an explicit test that patches RemoteBackendRegistry.create and asserts zero calls under default config.
• G2. Backward-compat shim. arctic_platform.rl.processors.verl_grpo continues to resolve to the moved implementation. Shim tested in Layer 2.
• G3. API-stability commitment. RemoteBackend ABC signature stays byte-identical to what @zw0610 approved on 2026-06-01. Any signature change requires a new PR.
• G4. Reproducible run scripts committed with the PR. arctic_platform/integrations/verl/examples/run_*.sh are the exact scripts used for Golden Runs.
• G5. Rollback plan. Pre-drafted single-commit revert PR sitting in draft on both repos. Package independence means arctic-platform / verl can be reverted separately.
• G6. Canary before broadcast. Post-merge Day 0: kick off 32B BIRD convergence run against merged code. Auto-alert on reward drop >10% below reference. No external announcement until canary hits 100 steps clean.

Risks & mitigations

Risk	Probability	Impact	Mitigation
V1 hook lifecycle differs from what we modeled (_compute_old_log_prob async in some subclass)	Medium	+3–5d rework	Day-1 design spike reads trainer_sync.py, trainer_colocate_async.py, trainer_separate_async.py before writing code
TransferQueue incompatible with sync asyncio.run bridge	Medium	+1–2d	Mock TransferQueue in Layer 1 unit tests catches this before smoke
LLMServerClient wants to own the rollout server; our replica hosts its own vLLM	Medium	+2–3d	Read verl/workers/rollout/llm_server.py in Day-1 spike; if incompatible, shim through LLMServerClient-compatible facade in rollout replica
CUDA IPC weight sync breaks silently (seen before with ARCTIC_WEIGHT_SYNC_STRICT_NAMES=0)	Low	Reward → 0	Layer 4 Golden Run 1 catches this — reward at step 4 vs step 1 must show learning
Xibin has additional design requests on the V1-port PR (like zw0610 originally)	Medium	+1 week per round	Day-1 design DM to preempt
CLA signature delays (currently 4 of 5 committers unsigned on PR #6422)	High	+1 week per delay	Chase this week, escalate to legal on Day 2 if stalled
verl-project/verl-recipe#116 (verl-tinker) merges first with conflicting V1-trainer primitives	Low	+1–2d realignment	Track weekly; align imports if it lands ahead
Downstream user imports arctic_platform.rl.processors.verl_grpo from unknown location	Low	External breakage	GitHub code search before releasing; keep shim indefinitely

Open questions (raise in the Xibin/Tunji/Zhi sync)

1. Recipe vs external-module? verl-project/verl-recipe/verl_tinker (landed 2026-07-03) uses the recipe pattern, not the VERL_USE_EXTERNAL_MODULES pattern. Would Xibin accept an arctic-platform-hosted plugin (this plan), or does he specifically want a verl-project/verl-recipe/arctic_rl/ recipe?
2. V1 trainer hook mapping. Are _compute_old_log_prob / _compute_ref_log_prob / _update_actor the right override points, or should we implement TrainingWorker / BaseEngine instead (the v0.7 blog described TrainingWorker as "Tinker-like")?
3. CLA path. Chase all 4 unsigned committers, or squash+re-author PR #6422 under @sfc-gh-truwase only (fastest merge)?

References

• Upstream RFC: verl-project/verl#6537 — needs status update reflecting this direction
• Upstream PR: verl-project/verl#6422 — the PR being restructured by this plan
• Related upstream work: verl-project/verl-recipe#116 (verl-tinker, filed 2026-07-03), verl-project/verl#4216 (Bedrock AgentCore RFC, closed), verl-project/verl#5737 (RemoteAgentLoop feature request)
• Prior Snowflake PRs (all merged): SAIR verl test full sha ci workflow #2, RL correctness ports #3, Basic import test #4, unbreak zorro #5, Fix zorro labels bug #6

/cc @sfc-gh-truwase @sfc-gh-sbekman