V1 (verl-project/verl main) plugin shims for the Arctic verl integration - #41
Open
sfc-gh-kganesan wants to merge 6 commits into
Open
V1 (verl-project/verl main) plugin shims for the Arctic verl integration#41sfc-gh-kganesan wants to merge 6 commits into
sfc-gh-kganesan wants to merge 6 commits into
Conversation
Adapts the Cortex backend for SkyRL and verl to the client/config refactor Mike landed in #75 (minimal-install) and #77 (onprem/remote as backend types, backend_config -> backend field, protocol renames). client/config: adds CortexConfig.from_env(), the single call-site framework adapters use to hydrate CortexConfig from ARCTIC_CORTEX_* env vars. Removes the previous scattered env-reads in the shim and the verl adapter. client/transports/cortex: wake/sleep short-circuit in the transport (_NOOP_OPS). Cortex sub-jobs are always awake; short-circuiting here means the shim doesn't have to wrap every wake/sleep — including the ones sync_weights invokes internally. rl/config, rl/client: legacy SkyRL config gets "cortex" backend + _backend_from_env validator (ARCTIC_BACKEND=cortex flips SkyRL's baked-in "local" without patching SkyRL); rl/client.create_arctic_rl_client dispatches to the shim lazily. rl/_cortex_dispatch: legacy -> unified config translation, async facade over ArcticRLClient, save_weights raises NotImplementedError (Cortex sub-jobs don't share disk; silent no-op would leave sampling on stale weights). integrations/_cortex_shared.to_cortex_fwd_bwd_payload: shared reshape between SkyRL shim and verl adapter (both build {batch, meta, processing}; Cortex takes {args, kwargs, context, processing}). integrations/skyrl: install_cortex_driver_shims patches SkyRL's peer_access_supported probe (hangs a CPU-only driver). Launcher python -m arctic_platform.integrations.skyrl installs it and forwards to SkyRL's own entrypoint. integrations/verl/adapter: ARCTIC_BACKEND=cortex swaps default OnPremConfig -> CortexConfig.from_env(); _zero_logprob_response fails loud on use_kl_loss / use_kl_in_reward / ppo_epochs > 1 (Cortex has no /forward, zero-fill is only correct for single-epoch on-policy GRPO without KL). Docs + tests: docs/cortex-integration.md, recipes/rl/skyrl/simple_gsm8k_cortex/, integrations/verl/examples/README-cortex.md + run_gsm8k_grpo_cortex.sh, tests/client/test_client_ops.py adds CortexConfig.from_env, transport noop-op, legacy env-promotion, shared-helper reshape, save_weights fail-loud coverage. Co-authored-by: Cursor <cursoragent@cursor.com>
…lure The server-side reason field disambiguates rate-limits, allowlist rejections, capacity exhaustion, and real sub-job crashes; today _wait_running throws 'reached terminal state failed' with no detail, which forces an out-of-band job query to make sense of it. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds V1-compatible wrappers for the Arctic verl backend so a single plugin install works against both Snowflake-AI-Research/verl (V0) and verl-project/verl main (V1). V0 modules are untouched. New: * integrations/verl/v1/worker.py: ArcticV1ActorRolloutRefWorker. Subclasses verl V1's ActorRolloutRefWorker; skips megatron/veomni init paths Arctic doesn't populate, drives async backend RPCs on a persistent background asyncio loop (compensates for @register losing coroutine-function status), and densifies V1 nested-jagged batches to V0's dense-padded layout via _to_v0_padded_batch. Uses CONFIG-level max_prompt_length / max_response_length (not batch-local maxes) so ZoRRo's Qwen3ModelOncePatcher can derive the correct prompt/response split. * integrations/verl/v1/replica.py: ArcticV1Replica. Adapts the V0 ArcticReplica constructor to V1's LLMServerManager.replica_init_kwargs forwarding and swaps in ArcticV1LLMServer. * integrations/verl/v1/server.py: ArcticV1LLMServer. Publishes TokenOutput.extra_fields['global_steps'] (V1's field; V0 used the now-silently-dropped extra_info kwarg) and exposes set_global_steps so the replica can fan the current policy version out after each CheckpointEngineManager weight sync. * integrations/verl/v1/examples/run_gsm8k_grpo_arl_v1.sh: V1-recipe smoke test. Updated: * integrations/verl/register.py: auto-detect V0 vs V1 based on whether verl.trainer.ppo.v1.trainer_remote_backend imports. Companion verl-core PR: https://github.com/verl-project/verl/pull/<TBD>. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Covers layout, source clones, env creation, install order (with the dss-client-before-arctic-training and torch-reinstall-after-flashinfer gotchas), data paths, launcher invocation, and the env vars the plugin keys on. Verified on 8xH200 with torch 2.10.0+cu129 / vllm 0.18.0 / flashinfer 0.6.6. Co-authored-by: Cursor <cursoragent@cursor.com>
sfc-gh-kganesan
force-pushed
the
karthik/verl-v1-plugin
branch
from
August 19, 2026 23:48
908326e to
583be50
Compare
Collaborator
Author
|
Rebased on latest CPU-only V1 seam smoke against the rebased verl PR (
GPU E2E on the rebased stack is queued (Cortex QA6 currently occupied by the PR#55 SkyRL run); will post GSM8K convergence numbers once the slot frees. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
V1 (verl-project/verl main) plugin shims for the Arctic verl integration
Summary
Adds V1-compatible wrappers so a single plugin install works against both
the Snowflake fork (V0,
Snowflake-AI-Research/verl@release/v0.7.1) andupstream (V1,
verl-project/verl@main). V0 modules are untouched;register.pyauto-detects which verl is installed based on whetherverl.trainer.ppo.v1.trainer_remote_backendimports.Companion verl-core PR (adds the V1 seam this plugin plugs into): verl-project/verl#7102
after the verl PR is opened._
New files
arctic_platform/integrations/verl/v1/worker.pyArcticV1ActorRolloutRefWorker, subclass of verl V1'sActorRolloutRefWorker. Highlights:_AsyncRunnerdrives the async backend RPCs on a persistentbackground asyncio loop (compensates for V1's
@registerdecoratorstack dropping coroutine-function status).
_to_v0_padded_batchdensifies V1's nested-jagged batches to V0'sdense-padded layout using config
max_prompt_length/max_response_length(not batch-local maxes) so ZoRRo'sQwen3ModelOncePatchercan derive the correct prompt/response split(
prompt_len = input_ids.shape[1] - max_response_lengthon everyforward — batch-local padding silently corrupts this).
arctic_platform/integrations/verl/v1/replica.py—ArcticV1Replicaadapts the V0
ArcticReplicaconstructor to V1'sLLMServerManager.replica_init_kwargsforwarding and points toArcticV1LLMServer.arctic_platform/integrations/verl/v1/server.py—ArcticV1LLMServerpublishesTokenOutput.extra_fields["global_steps"](V1's real field; V0 usedextra_infowhich pydantic silently drops) and adds aset_global_stepshook soCheckpointEngineManagercan fan thecurrent policy version out to servers after each weight sync.
arctic_platform/integrations/verl/v1/examples/run_gsm8k_grpo_arl_v1.shV1-recipe smoke launcher.
Updated files
arctic_platform/integrations/verl/register.py— auto-detects V0 vsV1 by trying to
import verl.trainer.ppo.v1.trainer_remote_backend.Validation
BIRD text-to-SQL, Qwen3-8B, 8× H200, recipe-aligned with
recipe/skyrl-integration/recipes/rl/verl/txt2sql/run_qwen3_32b_bird_grpo_arl_zorro_yes.sh,5 steps each.
timing_s/update_actortiming_s/gentiming_s/step(total)Convergence:
critic/score/meanmatches step-over-step between stock andArctic paths (step 1: 0.635 vs 0.635; step 5: 0.63 vs 0.63).
Known follow-up (not in this PR)
Inference (
gen) is currently at parity, not ahead. The pre-pluginrecipe forced
actor_rollout_ref.rollout.enforce_eager=True, which setscudagraph_mode=NONEand lets FCA fire (all 8 inference workers logForest Cascade Attention ENABLED (... cudagraph_mode=NONE)); with thatknob flipped,
gendrops to ~191 s (~7% inference speedup on 8B BIRD).The current plugin recipe (
recipe/skyrl-integration/.../run_qwen3_32b_bird_grpo_arl_zorro_yes.sh)uses
enforce_eager=False, and vLLM 0.18.0's new defaultcudagraph_mode=FULL_AND_PIECEWISEsilently disables FCA there. Fixingthis cleanly means having
parse_arctic_inference_rolloutinarctic_platform/rl/utils/server_models.pyoverridecompilation_config.cudagraph_mode=PIECEWISEwhenzorro_inference.enable=Trueandenforce_eager=False. Tracking as aseparate PR.