Cortex-serverless enablement for verl + SkyRL via the unified ArcticRLClient

[sfc-gh-kganesan]

Tracking issue for Cortex-serverless enablement of the two merged upstream RL integrations — NovaSky-AI/SkyRL#1837 (integrations/arctic_rl/) and verl-project/verl#6422 (Arctic-specific RemoteBackend in arctic_platform/integrations/verl/adapter.py) — without changing either integration's code.

Design principle

Both integrations construct their RL client via exactly one call:

from arctic_platform.rl import ArcticRLClientConfig, create_arctic_rl_client
client = create_arctic_rl_client(config, server_state)

Both hardcode backend="local" in that construction (SkyRL config.py:570, verl adapter.py:559), so a yaml-level backend=cortex is a silent no-op today. The bridge: an env-var override inside create_arctic_rl_client that rewrites the incoming config to Cortex when the launcher exports ARCTIC_RL_BACKEND=cortex. Zero adapter change.

Scope on the Arctic-Platform side (done or in flight)

1. Unified client + Cortex transport (Mike, done or in review)

• #45: arctic_platform.client.ArcticRLClient + Transport ABC + on-prem RayTransport.
• #46: on-prem HttpTransport.
• #47: CortexTransport + DSSST1 safetensors wire codec. Open, MERGEABLE, mergeStateStatus=BLOCKED. feat(rl): Cortex-serverless dispatch via arctic_platform.rl (zero integration change) #50 stacks on this.

2. Compat + arctic_platform.rl dispatch + env-var override + fake Cortex GS — #50 (draft)

Stacked on #47. All changes on the Arctic-Platform side; zero integration touch.

• Compat surface on arctic_platform.client (legacy backend aliases, response-shape flattening, colocation-lifecycle ops, **legacy_kwargs folding on fwd_bwd / fwd_no_grad, sync_weights(cuda_ipc, low_memory), training_job_id / sampling_job_id / log_prob_job_id / get_server_state properties). 26 pinning tests in tests/client/test_skyrl_verl_compat.py.
• Response-shape shim in CortexTransport._shape_train_response (aliases model_outputs → batch, mirrors loss ↔ avg_loss, bubbles metrics.grad_norm to top level so .get("grad_norm") works uniformly across on-prem and Cortex).
• On-prem parity fields threaded through the unified init (ds_worker_config, arctic_inference_config, log_prob_ds_config, full_determinism).
• RayTransport accepts a server_state= parameter so verl's driver → Ray-worker reconnect flow keeps working.
• arctic_platform.rl.create_arctic_rl_client(config, server_state) dispatches to Cortex when config.backend == "cortex": translates to a unified config, constructs an arctic_platform.client.ArcticRLClient, and wraps it in _CortexClientShim that re-exposes the exact async surface both integrations reach for. 29 pinning tests in tests/client/test_rl_cortex_dispatch.py.
• Env-var override in the same factory. Both adapters hardcode backend="local"; the launcher exports ARCTIC_RL_BACKEND=cortex + CORTEX_* and the factory rewrites the config before dispatch. 6 tests under TestEnvOverride.
• Lazy arctic_platform/rl/__init__.py (PEP 562 __getattr__). Cortex users install only arctic-platform + pydantic + requests; no ray / vllm / arctic_inference / torch required on the driver.
• Fake Cortex GS + 18-test E2E plumbing smoke at tests/e2e/. Speaks every REST route the client's CortexTransport actually hits, decodes DSSST1 chunked uploads, returns shape-plausible canned responses (random losses, shape-correct logprobs tensors). Doubles as an executable interface reference for the real Cortex-training endpoint. Total 116 tests pass locally on a driver with only pydantic + requests + fastapi + uvicorn + safetensors + torch installed.

3. Cortex server-side (Neutrino GS team) — blocking real Cortex training convergence

The client PR ships shims (log a None for missing metrics.grad_norm, _colo_noop for lifecycle ops Cortex doesn't have) so nothing crashes when these are absent, but a real Cortex-backed training step needs:

• POST /{job_id}/forward-no-grad — symmetric to forward-backward. Accept {args, kwargs, meta, processing, reference_model?} (DSSST1 chunks in) and return {model_outputs: {logprobs, entropy?}, metrics: {...}} (DSSST1 out). verl calls this twice per batch (actor + ref); SkyRL calls it in the logprobs post-processor path. Runnable spec: fake_cortex_gs.py.
• POST /{job_id}/log-probs — JSON in / DSSST1 out. Symmetric to generate in framing. Optional for verl; blocking for any recipe that scores completions server-side.
• Grow metrics: {grad_norm, ppo_kl, pg_clipfrac_lower, pg_loss, kl_loss, kl_coef} on fwd_bwd / step responses. Client currently None-shims these; both integrations reduce them into wandb.
• Batch contract validation: Cortex Neutrino trainer proto is additionalProperties: true, so pre-tokenized kwargs={input_ids, attention_mask, position_ids, prompts, responses, response_mask, advantages, old_log_probs, ref_log_prob?} should round-trip today. Needs an end-to-end call on a real cluster to catch any silently-dropped keys.

4. Full E2E validation (parallel-track, needs both sides)

• On-prem regression smoke — run arctic_platform/integrations/verl/examples/run_gsm8k_grpo_arl.sh unmodified with backend: local (default) on 1× H200. Loss curves should match main within numerical noise. Only relevant if we're worried about feat(rl): Cortex-serverless dispatch via arctic_platform.rl (zero integration change) #50 perturbing the default path; the unit tests cover the surface but there's no substitute for a real training run. Requires verl + arctic-inference[vllm] + tensordict installed.
• Cortex first run — same recipe with ARCTIC_RL_BACKEND=cortex + a real Cortex endpoint. Unlocks once §3 items land.

What we deliberately are not doing

• No integration-side patches. Neither NovaSky-AI/SkyRL#1837:integrations/arctic_rl/ nor arctic_platform/integrations/verl/adapter.py gets any import swap, await restructure, or config schema change.
• No fork of arctic_platform.rl. The legacy on-prem client stays as-is; the Cortex path lives behind a single early-branch in create_arctic_rl_client plus the env-var override.
• No dep changes for on-prem users. The lazy __init__ refactor keeps every arctic_platform.rl.* export addressable; the only difference is when the underlying module is loaded. Existing test_cpu_import.py still passes.

User-facing UX (what the two integrations flip to go serverless)

SkyRL — set these in the recipe launcher (existing script, no code change):

export ARCTIC_RL_BACKEND=cortex
export CORTEX_BASE_URL=https://cortex.snowflakecomputing.com   # or staging / mock
export CORTEX_DATABASE=my_db
export CORTEX_SCHEMA=rl
export CORTEX_ENDPOINT=cortex-training
export CORTEX_PAT_ENV_VAR=CORTEX_PAT
bash integrations/arctic_rl/examples/run_gsm8k_grpo_4gpu.sh

verl — same env-var pattern, same launcher:

export ARCTIC_RL_BACKEND=cortex
export CORTEX_BASE_URL=...
bash arctic_platform/integrations/verl/examples/run_gsm8k_grpo_arl.sh

Neither adapter's yaml needs a new field, and both continue to hardcode backend="local" internally — the factory rewrites it.

Rollout order

1. feat(client): add Cortex transport + DSSST1 wire codec #47 merges (Mike).
2. feat(rl): Cortex-serverless dispatch via arctic_platform.rl (zero integration change) #50 merges. Both integrations gain Cortex support the moment feat(rl): Cortex-serverless dispatch via arctic_platform.rl (zero integration change) #50 lands — no follow-up PRs needed on either integration.
3. Once §3 endpoints land on GS, flip a recipe launcher's env to ARCTIC_RL_BACKEND=cortex for the first end-to-end serverless smoke.

References

• Compat + dispatch + env-override + fake GS PR: #50
• Runnable Cortex interface reference: tests/e2e/fake_cortex_gs.py on the PR branch
• verl adapter (unchanged): arctic_platform/integrations/verl/adapter.py
• SkyRL integration (unchanged): NovaSky-AI/SkyRL:integrations/arctic_rl/
• Pinning tests: tests/client/test_skyrl_verl_compat.py + tests/client/test_rl_cortex_dispatch.py + tests/e2e/test_cortex_transport_smoke.py
• Unification notes: arctic_platform/client/UNIFICATION_NOTES.md

Superseded / closed

• #51 — verl adapter swap. Closed. Replaced by the env-var override + arctic_platform.rl dispatch shim in feat(rl): Cortex-serverless dispatch via arctic_platform.rl (zero integration change) #50; both adapters remain unchanged.

[sfc-gh-kganesan]

Compat draft PR up: #50 (stacked on #47). Merge order: #47 → #50 → verl adapter swap → SkyRL adapter swap.

[sfc-gh-truwase]

For user-facing UX, please prefer yaml changes and avoid env vars.