[arctic-rl] align adapter with upcoming zorro_train config grouping by sfc-gh-kganesan · Pull Request #5 · Snowflake-AI-Research/verl

sfc-gh-kganesan · 2026-06-09T03:33:31Z

Summary

Aligns the Arctic remote-backend adapter with the upcoming arctic_training API rename + adds opt-in CUDA IPC weight sync. Two-commit branch; both adapter-internal, both validated end-to-end against the post-rename upstream stack.

Stacked on top of the public abstract-backend PR head (arctic_rl_share_v0.7.1 @ cbf9cd5a). This branch is the staging ground for the follow-up upstream PR we'll open against verl-project/verl:main once verl-project#6422 merges there.

Commits

Commit	Summary
`21ac6e8d` + `394e6fb3`	`use_zorro` → `zorro_train.enable` rename; new `arctic_training.rl` import path; `checkpoint_path` plumbed into `ArcticRLClientConfig`; `zorro_train_max_rollouts` forwarded in per-call meta
`a5323c9f`	`cuda_ipc_weight_sync` + `low_memory_weight_sync` plumbing; `async_generate` → `generate(..., routing_key=None)` to match post-rename arctic_training API

Scope

4 files, all adapter-internal. No changes to the RemoteBackend ABC, registry, generic worker/trainer, or any non-Arctic code.

File	Change
`verl/trainer/config/remote_backend/arctic.yaml`	(a) Replace flat `use_zorro: False` with nested `zorro_train: { enable, max_rollouts }`. (b) Add `cuda_ipc_weight_sync: False` and `low_memory_weight_sync: False` (both opt-in, defaults preserve current NCCL path)
`verl/workers/remote_client/arctic_rl.py`	(a) Rename `self.use_zorro` → `self.zorro_train_enable`. (b) Update 2 `meta=dict(...)` payloads + `ds_worker_config.update(...)` kwarg. (c) Update import path to `arctic_training.rl`. (d) Pass `checkpoint_path`. (e) Read `cuda_ipc_weight_sync` / `low_memory_weight_sync` from config and forward to `_client.sync_weights(cuda_ipc=, low_memory=)`. (f) Rename `async_generate` → `generate(..., routing_key=None)`
`examples/arctic_rl/run_gsm8k_grpo_arl_zorro_yes.sh`	Update Hydra override to `remote_backend.zorro_train.enable=True`

Validation

Local CI (pre-commit, matches verl's CI hooks)

All 7 hooks pass on the modified files:

ruff (legacy alias).............................................................Passed
ruff format.....................................................................Passed
mypy............................................................................Passed
Generate and verify verl/trainer/config/_generated_*.yaml.......................Passed
Check doc string coverage.......................................................Passed
Check license...................................................................Passed
Compile all python files........................................................Passed

E2E smoke: 4/4 passed against post-rename stack

4-step GSM8K-GRPO smoke completed end-to-end with colocate=True + cuda_ipc_weight_sync=True (CUDA IPC weight-sync path; no NCCL handshake required):

Step	actor/loss	grad_norm	MFU	score	update_weights (s)	update_actor (s)
1	0.082	0.361	211	0.250	0.82	3.47
2	0.093	0.331	246	0.213	1.10	3.26
3	0.055	0.316	219	0.188	1.22	3.33
4	0.129	0.355	220	0.263	1.03	3.39

Training Progress: 100%|██████████| 4/4 [02:21<00:00, 35.37s/it]

CUDA IPC path confirmed active via log line:
[weight-sync names validated] context=cuda_ipc sender=310 expected=310

Stack used for E2E:

arctic_training @ tunji/verl_integration HEAD (post-rename)
arctic_inference @ mert/verl_integration_vllm18 (vLLM 0.18 compatible)
vllm==0.18.0, torch==2.10.0+cu129, flashinfer-python==0.5.3
flash_attn 2.8.3 + flash_attn_3 3.0.0 (H200 wheels for torch 2.10)

A longer convergence run is in progress; will attach the log/curve when done.

Why this matters

Without the cuda_ipc_weight_sync path the adapter falls back to NCCL all_reduce, which on our reproducer setup hangs indefinitely at stateless_init_nccl during the first sync_weights (both ends stuck in ncclAllReduce inside PyNcclCommunicator.__init__). The CUDA IPC path is zero-copy via cudaIpcGet/OpenMemHandle and avoids the NCCL bootstrap entirely — sub-1.3s sync times observed on Qwen3-0.6B vs the previous indefinite hang.

Defaults are preserved (cuda_ipc_weight_sync: False, low_memory_weight_sync: False); existing configs are unaffected.

Notes for reviewers

The yaml comments document why the adapter forwards enable as a flat zorro_train_enable kwarg rather than the nested dict — upstream server reads it flat, yaml nesting is for config organization.
checkpoint_path defaults to self.config.trainer.default_local_dir, which verl always sets — no caller breakage.
routing_key defaults to None, preserving behaviour for callers that haven't migrated to Mert's FCA group routing.
Mirrors the canonical reference run at examples/arctic_rl/run_big_bird_grpo_arl_zorro_yes.sh (which uses the same colocate=True + cuda_ipc_weight_sync=True combo).

Co-Authored-By: Cursor cursoragent@cursor.com

Tracks an upcoming arctic_training API change that regroups `use_zorro` into a nested `zorro_train` block. Changes (3 files, +21/-10): * `verl/trainer/config/remote_backend/arctic.yaml` - replace flat `use_zorro: False` with nested `zorro_train: { enable: False, max_rollouts: <rollout.n> }` * `verl/workers/remote_client/arctic_rl.py` - rename `self.use_zorro` -> `self.zorro_train_enable` - rename `meta=dict(use_zorro=...)` -> `meta=dict(zorro_train_enable=...)` in `_build_log_prob_payload` and `_build_update_actor_payload` - update `ds_worker_config.update(use_zorro=True)` -> `zorro_train_enable=True` - update import path: `arctic_training.arctic_rl` -> `arctic_training.rl` - pass `checkpoint_path=self.config.trainer.default_local_dir` into `ArcticRLClientConfig` (now required for training jobs) * `examples/arctic_rl/run_gsm8k_grpo_arl_zorro_yes.sh` - update Hydra override: `remote_backend.use_zorro=True` -> `remote_backend.zorro_train.enable=True` Why now: the upstream server now reads `zorro_train_enable` from `ds_worker_config` and from per-call request metas. Without this rename, the adapter silently disables ZoRRO (the missing key defaults to False) once users upgrade arctic_training. Validation (local): * pre-commit (ruff, ruff-format, mypy, autogen-trainer-cfg, check-docstring-coverage, check-license, compileall) - 7/7 Passed * import test against post-rename arctic_training - imports succeed * ArcticRLClientConfig pydantic v2 schema validation - all kwargs accepted * Ray cluster bootstrap + dataset load + TaskRunner spin-up - OK * Full 4-step GSM8K E2E - blocked on an unrelated arctic_inference version skew (ReplicaPool.reset_prefix_cache missing on the locally installed arctic_inference); will re-validate once a coordinated arctic_training + arctic_inference release lands. Targets: this PR is stacked on the public abstract-backend PR (arctic_rl_share_v0.7.1). When that lands in verl-project/verl, rebase + open the equivalent upstream follow-up. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Karthik Ganesan <karthik.ganesan@snowflake.com>

Followup to the zorro_train rename: the yaml exposes `zorro_train.max_rollouts` but the adapter never forwarded it to the per-call meta dicts, so users could set it in config and it would silently have no effect (upstream falls back to `rollout_n` via `meta_data.get("zorro_train_max_rollouts", meta_data["rollout_n"])` in `arctic_training/rl/utils/batch.py`). Add the field on `__init__` and pass it alongside `zorro_train_enable` in both `_build_log_prob_payload` and `_build_update_actor_payload`. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Karthik Ganesan <karthik.ganesan@snowflake.com>

Two adapter-side changes needed to E2E-validate against post-rename arctic_training: 1. cuda_ipc / low_memory weight-sync plumbing Adds `cuda_ipc_weight_sync` and `low_memory_weight_sync` keys to `remote_backend/arctic.yaml` (both default False, opt-in) and forwards them on every `update_weights` call to `arctic_training.rl.ray_client.sync_weights(cuda_ipc=, low_memory=)`. With `colocate=True + cuda_ipc_weight_sync=True`, weight sync uses zero-copy CUDA IPC (cudaIpcGet/OpenMemHandle) instead of NCCL all_reduce, bypassing the NCCL bootstrap entirely. Sub-1.3s sync times observed on Qwen3-0.6B vs the previous NCCL-bootstrap hang. 2. async_generate -> generate API rename arctic_training renamed the rollout-side method from `async_generate(...)` to `generate(..., routing_key=None)` (the new kwarg threads through to Mert's FCA group routing). Our wrapper forwards the kwarg as-is; defaulting to None preserves existing behaviour for callers that don't set it. E2E proof: 4-step GSM8K-GRPO smoke completed 4/4 with cuda IPC weight sync confirmed via `[weight-sync names validated] context=cuda_ipc` log line; MFU 211-246, update_weights 0.8-1.2s. Signed-off-by: Karthik Ganesan <karthik.ganesan@snowflake.com> Co-authored-by: Cursor <cursoragent@cursor.com>

sfc-gh-kganesan requested review from sfc-gh-sbekman and sfc-gh-truwase June 9, 2026 04:12

sfc-gh-kganesan and others added 2 commits June 9, 2026 17:57

sfc-gh-sbekman approved these changes Jun 10, 2026

View reviewed changes

sfc-gh-kganesan marked this pull request as ready for review June 10, 2026 23:21

sfc-gh-kganesan merged commit 2174486 into arctic_rl_share_v0.7.1 Jun 10, 2026
1 check passed

Navigation Menu

Uh oh!

[arctic-rl] align adapter with upcoming zorro_train config grouping - #5

[arctic-rl] align adapter with upcoming zorro_train config grouping#5
sfc-gh-kganesan merged 3 commits into
arctic_rl_share_v0.7.1from
karthik/zorro-train-rename

sfc-gh-kganesan commented Jun 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sfc-gh-kganesan commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Commits

Scope

Validation

Local CI (pre-commit, matches verl's CI hooks)

E2E smoke: 4/4 passed against post-rename stack

Why this matters

Notes for reviewers

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sfc-gh-kganesan commented Jun 9, 2026 •

edited

Loading