harbor: Cortex training backend as a Harbor plugin (draft) by sfc-gh-kganesan · Pull Request #66 · Snowflake-AI-Research/Arctic-Platform

sfc-gh-kganesan · 2026-08-10T21:44:32Z

Draft. Stacks on top of #55 (Cortex dispatch shim + SkyRL/verl+Cortex recipes) — please merge that first, then re-target this PR to main.

What this PR does

Two things:

Adds arctic_platform/integrations/harbor/ — a Harbor plugin that trains models end-to-end using Harbor's own CLI as the driver, with Cortex Training as the RL backend. Same in-tree subpackage shape as arctic_platform/integrations/verl/.
Adds an OpenAI-compatible HTTP surface (/v1/chat/completions, /v1/completions, /v1/models) to the sampling sub-job's existing FastAPI server, so any Harbor BaseAgent that speaks OpenAI-chat (Terminus 2 via LiteLLM, LangChain-backed agents, custom BaseAgents) can drive rollouts without RL-specific plumbing.

Direction of integration: Harbor is the primary product; Arctic ships a plugin for it. Harbor discovers the plugin via importlib.metadata entry points registered on the top-level pyproject.toml:

$ harbor plugins list
┃ Name                ┃ Import path                                             ┃
│ arctic-cortex-agent │ arctic_platform.integrations.harbor.agent:CortexRLAgent │
│ arctic-cortex-env   │ arctic_platform.integrations.harbor.env:HostEnvironment │

Every LLM call runs inside a harbor run trial (BaseAgent under BaseEnvironment, RolloutDetail written to disk), scored by Harbor's stock Verifier execing each task's tests/test.sh. Between trials, the driver reads Harbor's result.json, runs one Cortex GRPO step, and sync_weights propagates the new weights back to the same sampling sub-job — so the next harbor run samples from an improved model at the same endpoint. No Harbor code is modified. No custom BaseVerifier.

Two sampling modes (both working E2E)

Mode	Agent	How it reaches the sampling sub-job
Native (default)	`CortexRLAgent` (or any agent consuming `ArcticRLClient.reconnect_config`)	RL-shaped `/generate` route, DSSST1 octet wire.
OpenAI-compat (`--sampling-api-base`)	Any Harbor `BaseAgent` that speaks OpenAI-chat — Terminus 2, LangChain-backed agents, custom `BaseAgent`s	`/v1/chat/completions` / `/v1/completions` / `/v1/models` on the same sub-job, translated into `ReplicaPool.generate()` in-process.

Both routes share the same worker pool: /generate and /v1/* route through the same ReplicaPool, so scheduling / prefix cache / TP fan-out are identical.

End-to-end demo: Terminus 2 training on Cortex, on-prem

# 1. Bring up the RL server (one training + one sampling GPU).
python -m arctic_platform.rl.http_server \
  --training-gpus 1 --sampling-gpus 1 --port 7000 &

# 2. Drive training with Harbor's stock Terminus 2 agent over OpenAI-compat.
harbor-cortex-train \
  --tasks-dir ./my_bench/train  --heldout-dir ./my_bench/heldout \
  --model             Qwen/Qwen3-0.6B \
  --agent             harbor.agents.terminus_2:Terminus2 \
  --sampling-api-base auto \
  --llm-backend       litellm \
  --iters 30 --prompts-per-step 8 --n-attempts 4 --lr 5e-6 \
  --out ./training-run/

--sampling-api-base auto reads the client's transport to derive the URL (on-prem: http://localhost:7000/v1). On Cortex, once sub-job URL routing lands at the ingress, users pass the sub-job URL explicitly — this repo does not need any further changes.

What's in the package

`arctic_platform/openai_compat.py` (new; 605 LOC)

Thin OpenAI-compat router mounted on arctic_platform.rl.http_server.

/v1/chat/completions: renders messages through the tokenizer's chat template (loaded once at /initialize), calls pool.generate, formats the response. Supports n>1, stop, max_completion_tokens, presence_penalty/frequency_penalty, seed, stream, logprobs.
/v1/completions: string prompt, list[int] token prompt, or batched list[list[int]].
/v1/models: returns the currently-loaded sampling model; empty list (200, not 503) before init so LiteLLM's capability probe doesn't give up.
stream=True: runs the full generation and replays as SSE deltas in OpenAI's chunked shape. Client contract is correct; first-token latency is not (needs a delta-yielding surface on ReplicaPool — noted in PLAN.md).
Unknown OpenAI fields (tools, response_format, parallel_tool_calls, ...) are accepted and ignored so newer SDK versions don't 422.
Lives above arctic_platform.rl/ so tests can import the router without pulling the training kernel (tensordict, Ray, DeepSpeed).

`arctic_platform/rl/http_server.py`

Mounts openai_compat.router alongside /generate.
/initialize (sampling branch) now loads the tokenizer once and stores it on app.state.sampling_tokenizer / app.state.sampling_model_name.
/destroy clears the tokenizer + model_name on sampling teardown.

Harbor plugin (`arctic_platform/integrations/harbor/`)

File	Role
`models.py`	RFC data contract — `Rollout` (with `loss_mask`), `RolloutDataset`, `PostTrainingConfig`, `TrainingRun`, `InferenceEndpoint`. Mirrors Harbor's `RolloutDetail` 1:1.
`backend.py`	`ArcticCortexBackend` — RFC `PostTrainingBackend` protocol over `ArcticRLClient`. Owns GRPO advantage + batch construction.
`env.py`	`HostEnvironment(BaseEnvironment)` — host subprocesses under a per-trial root, no container. Development-only.
`agent.py`	`CortexRLAgent(BaseAgent)` — reference agent for the native reconnect-config path.
`adapter.py`	`_flatten_multi_turn` walks per-turn tokens into a flat `(prompt, completion, loss_mask)`. Marks every model-produced position across all turns as trainable; falls back to final-completion-only when the invariant fails.
`task_gen.py`	Arithmetic task-dir writer for the demo.
`train.py`	`harbor-cortex-train` CLI. Two sampling modes; `--sampling-api-base auto` derives the URL from the client's transport.
`aggregate.py`	`harbor-cortex-aggregate` — multi-seed aggregator with bootstrap 95% CI.
`PLAN.md`	Where every piece stands + what's outside this repo.
`RUN_LOG.md`	Transcript of the 3-seed experiment below.

Tests

File	Coverage
`tests/openai_compat/test_openai_compat.py`	19 unit tests. Parameter mapping (`temperature`, `top_p`, `top_k`, `seed`, `stop`, penalties, `max_completion_tokens`), `n>1` fan-out, streaming SSE structure (role → content deltas → finish frame → `[DONE]`), 503 before init, 400 for no chat template, 400 for bad body, finish-reason mapping (`length`, `abort → content_filter`, `None → stop`), token-id prompts, batched string / list-of-list-int prompts, unknown-field tolerance.
`tests/openai_compat/test_real_openai_sdk.py`	4 contract tests. Spins up a live `uvicorn` subprocess and drives it with the actual `openai` Python client: chat completions non-stream + stream, completions, `models.list`. Skips cleanly when `openai` isn't installed.
`tests/integrations/harbor/test_adapter_multiturn.py`	6 tests for the multi-turn flatten.

All 29 tests pass locally.

Experiment: 15-step GRPO on Cortex (native mode)

Qwen/Qwen3-0.6B, 3-digit × 2-digit multiplication (a ∈ [100, 999], b ∈ [10, 99]), 15 GRPO steps × 24 rollouts/step (6 prompts × 4 attempts), lr=5e-6, temp=0.8. Held-out 80 problems, greedy re-eval. Three seeds. Full transcript in arctic_platform/integrations/harbor/RUN_LOG.md.

                seed 0            seed 1            seed 2            n=3 aggregate
pass@1          0.362 → 0.350     0.350 → 0.400     0.375 → 0.425     Δ +0.029  95% CI [-0.013, +0.050]
mean held-out r 0.580 → 0.696     0.600 → 0.690     0.648 → 0.741     Δ +0.100  95% CI [+0.090, +0.116]

pass@1 CI spans zero: a 0.6B model doesn't reliably fix 3-digit × 2-digit multiplication in 15 GRPO steps.
Mean held-out reward moves +10.0 pp, 95% CI [+9.0, +11.6] across three seeds. On seed 0, 19 held-out problems moved out of the reward ≤ 0.05 bucket into "close but wrong" or "verbose correct".

Non-obvious client-side bugs found and fixed during bringup

ArcticRLClientConfig.training_job_id/sampling_job_id are Field(exclude=True) — model_dump_json() drops them. Each Harbor trial subprocess was cold-starting a fresh Cortex job and blocking on _wait_running. Fix in train.py: write ids back after dump.
ArcticRLClient.shutdown() cancels the shared Cortex sub-job. With one trial per subprocess, the first trial killed the job the rest needed. Fix in agent.py: drop shutdown() from the agent; the runner cancels in its finally.
AutoTokenizer.from_pretrained on every trial hits HF Hub → rate limit → dropped trials. Fix: pre-warm cache in the runner, local_files_only=True in the agent.

What's still outside this repo

Only one item: Cortex control-plane routing so a client can reach https://<cortex>/sub-jobs/<sampling-sub-job-id>/v1/* from outside. The sub-job process already answers on /v1/* (this PR), and on-prem the URL is just http://<host>:<port>/v1. Once the ingress rule is in, no code here changes.

Non-blocking follow-ups (see PLAN.md):

harbor train subcommand upstream in Harbor — ~30-LOC Typer addition, dispatch through a harbor.backends entry point. arctic_platform.integrations.harbor.train:cli is a drop-in.
PostTrainingBackend.stream_progress() — async iterator around backend.train().
Real sandbox in production runs — swap HostEnvironment for Harbor's DockerEnvironment / ModalEnvironment / DaytonaEnvironment.

Sanitized for a public PR

Personal absolute paths stripped from RUN_LOG.md. Cortex sub-job UUIDs are invalidated (jobs shut down) and left in as evidence that baseline + final used the same sub-job. No PATs or internal hostnames in the diff.

Companion RFC

rfcs/harbor-post-training-backend.md (working tree; not merged). This PR is the reference implementation.

Made with Cursor

Co-authored-by: Cursor <cursoragent@cursor.com>

Adds a small async shim on top of the unified sync ArcticRLClient so legacy SkyRL / verl adapters (which construct their client via arctic_platform.rl.create_arctic_rl_client and expect an async surface) can drive Cortex unmodified. - arctic_platform/rl/_cortex_dispatch.py: _CortexClientShim wraps ArcticRLClient. _to_unified_config maps the legacy flat config onto the nested `backend_config: CortexConfig` + training / sampling layout from PR #54. fwd_bwd translates SkyRL/verl payloads onto Cortex's canonical {args, kwargs, context, processing} shape; fwd_no_grad returns [B, T] zero placeholders since Cortex has no /forward endpoint (the server grpo loss defaults to logprobs.detach() when old_log_probs_shifted is absent, so π_old ≡ π_new -- correct for single-epoch on-policy PPO). Once PR #58 (async unified client) merges, the `async def` wrappers here collapse to `return await self._client.foo(...)`. - arctic_platform/rl/client.py: route through the shim when config.backend == "cortex" or ARCTIC_BACKEND=cortex, so adapters that hard-code backend="local" work without patches. Lazy-import HTTP/Ray clients so CPU-only Cortex drivers don't pay the arctic_inference cost. - arctic_platform/rl/config.py: add "cortex" to the backend literal + the cortex_* endpoint fields; loosen job_id fields to Any (Cortex uses "<uuid>:training:0" tokens where on-prem stamps ints); skip _derive_host_port on the cortex branch. Cold-start default bumped to 1800s to match arctic_platform.client.config. - arctic_platform/rl/__init__.py: SkyRL's prepare_runtime_environment probes cudaCanAccessPeer via a {"CPU":1,"GPU":2} placement group. On a CPU-only Cortex driver that hangs forever. Monkey-patch peer_access_supported -> False under ARCTIC_BACKEND=cortex. - arctic_platform/integrations/verl/rollout.py: coalesce N concurrent per-prompt Ray fan-out calls into one batched Cortex /generate (~50 ms latch) — cuts a step's rollout from ~15 min to ~19 s on Cortex. - recipes/rl/skyrl/{simple_gsm8k,txt2sql,long_context_qa}_cortex/, arctic_platform/integrations/verl/examples/{README-cortex.md, run_gsm8k_grpo_cortex.sh}: Cortex siblings of the on-prem recipes. Verified end-to-end on Cortex-training QA6 with SkyRL GSM8K GRPO (Qwen/Qwen3-0.6B, 4T + 4S, 1 epoch, 116 steps ~1h) and verl GSM8K GRPO (4T + 4S, 40 steps, ~15m; loss bounded, reward 0.16 -> 0.35, entropy stable). Zero SkyRL / verl adapter patches required. Co-authored-by: Cursor <cursoragent@cursor.com>

…es on multi-element) Co-authored-by: Cursor <cursoragent@cursor.com>

…utDetail Adds arctic_platform/integrations/harbor/, a reference implementation of the Harbor Post-Training RFC driven by Harbor's own trial runner. Every LLM call happens inside a real `harbor run` (BaseAgent, BaseEnvironment, BaseVerifier, result.json). The adapter reads Harbor's on-disk RolloutDetail, hands it to ArcticCortexBackend.train on Cortex QA6, and sync_weights makes the next `harbor run` sample from the updated model at the same sub-job endpoint. Pieces: - backend.py: ArcticCortexBackend (RFC protocol) over ArcticRLClient + Cortex transport. Owns GRPO advantage + batch construction. - models.py: RFC data contract (Rollout, RolloutDataset, ...). - host_environment.py: BaseEnvironment on host subprocesses so Harbor runs on a box without Docker/Modal/Daytona. Not for prod. - cortex_agent.py: BaseAgent that reattaches to a running Cortex sub-job via ArcticRLClient.reconnect_config and writes RolloutDetail. - arithmetic_verifier.py: BaseVerifier — last-int extraction, comma-normalized, dense partial credit by relative error (GRPO always has gradient even when no rollout is exactly right). - adapter.py: Harbor result.json -> RolloutDataset. - task_gen.py: Programmatic Harbor task-dir + dataset.toml writer. - harbor_runner.py: End-to-end driver. real harbor run baseline -> N x (harbor run rollouts + backend.train) -> real harbor run post-training re-eval on same endpoint. E2E on Cortex QA6 (Qwen/Qwen3-0.6B, 3-digit x 2-digit MUL, 8 GRPO steps, 24 rollouts/step, lr=5e-6): baseline pass@1 0.250 -> final 0.312 (+0.062). Training reward curve climbs 0.375 -> 0.960 peak with real gradients throughout. Same sampling endpoint before and after. Full transcript + side-by-side completions in RUN_LOG.md. Co-authored-by: Cursor <cursoragent@cursor.com>

Three held-out problems flip wrong->right, zero regressions: 965 x 22: 21630 -> 21230 (actual 21230) 376 x 13: 4908 -> 4888 (actual 4888) 991 x 96: 95184 -> 95136 (actual 95136) Real arithmetic changes, not formatting. Full transcript, per-step gradients, and all 20 baseline / final completions in RUN_LOG.md. Co-authored-by: Cursor <cursoragent@cursor.com>

…; multi-seed aggregation Replaces the custom ArithmeticVerifier (BaseVerifier subclass) with Harbor's default ``harbor.verifier.verifier:Verifier``, which uploads each task's ``tests/test.sh`` and reads reward from ``/logs/verifier/reward.txt``. This is Harbor's canonical verifier path — exactly how a real Harbor benchmark task scores. No custom BaseVerifier, no ``--verifier`` override. Also: - Fix HF Hub rate-limit: warm tokenizer cache in runner, use ``local_files_only=True`` in CortexRLAgent. Was silently dropping 21/24 trials on step 11 with ``HfHubHTTPError``. - Adapter: preserve every verifier reward field on the rollout metadata so eval can report multiple metrics without re-reading result.json. - Runner: report both pass@1 and mean held-out reward, ``--seed`` for independent runs, extended summary.json. - Add ``aggregate_runs.py``: read N summary.jsons, print per-seed table + aggregate mean±sd + bootstrap 95% CI on the deltas. - Drop ``arithmetic_verifier.py`` — superseded by real test.sh. - README/RUN_LOG rewritten around the honest 2-seed result: pass@1 doesn't move (0.6B model can't fix arithmetic in 15 steps); mean held-out reward moves +10.3 pp with 95%CI [+9.0, +11.6] — real, statistically clean improvement in output quality (fewer catastrophic failures, closer answers). Two seeds, 15 GRPO steps × 24 rollouts, 80 held-out on Qwen3-0.6B: pass@1 seed0: 0.362 -> 0.350 seed1: 0.350 -> 0.400 aggregate Δ +0.019, 95%CI [-0.013, +0.050] mean held-out r seed0: 0.580 -> 0.696 seed1: 0.600 -> 0.690 aggregate Δ +0.103, 95%CI [+0.090, +0.116] Co-authored-by: Cursor <cursoragent@cursor.com>

Three independent seeds on 3-digit x 2-digit MUL, 15 GRPO steps x 24 rollouts, 80 held-out. All three seeds move in the same direction on both metrics. pass@1 baseline 0.362 +/- 0.010 -> final 0.392 +/- 0.031 delta +0.029 +/- 0.029, 95%CI [-0.013, +0.050] mean held-out r baseline 0.609 +/- 0.029 -> final 0.709 +/- 0.023 delta +0.100 +/- 0.011, 95%CI [+0.090, +0.116] Mean held-out reward has a tight positive CI even with n=3 — the demo-worthy metric. Pass@1's CI still spans zero, matching the honest framing in the README (0.6B model + 15 steps of RL doesn't fix arithmetic). Co-authored-by: Cursor <cursoragent@cursor.com>

…flow The runner previously baked in an arithmetic task generator, so a Harbor user with their own task directories couldn't drive it against Cortex without editing our code. This wires up the missing knobs on top of the same driver logic: --tasks-dir DIR Harbor dataset dir (or dir of task subdirs). Each step samples --prompts-per-step tasks and writes a per-step dataset.toml pointing at them. --heldout-dir DIR Held-out Harbor dataset for baseline + final eval. --skill-md PATH Passed to harbor run --extra-instruction-path (appended to every task's instruction.md). --skill-dir PATH Passed to harbor run --skill. --agent MOD:CLS Override the default CortexRLAgent (user's own BaseAgent that samples via ArcticRLClient works). --env MOD:CLS Override HostEnvironment (swap for DockerEnvironment in prod). --out DIR Alias for --work-dir. summary.json + harbor_jobs/ + reconnect_config.json land here. When --tasks-dir / --heldout-dir are omitted, the arithmetic generator still runs (the 3-seed demo reproduces byte-for-byte). summary.json now captures the model, agent, env, skill paths, task pool paths, and reconnect_config_path so downstream evals / prod inference can point at the same trained sub-job. README rewritten with a "Harbor user's flow" section up front: - Single command line for the E2E training run. - Table mapping each of the user's artifacts (task.toml, instruction.md, tests/test.sh, SKILL.md, skills/, custom BaseAgent) to Harbor's own extension surfaces — no bespoke flag translation. - Post-training re-use pattern: reconnect to the trained sub-job for further Harbor evals or production traffic. Co-authored-by: Cursor <cursoragent@cursor.com>

…ntegration) Direction of integration: Harbor is the primary product. This work ships as a Harbor plugin discovered through Harbor's own ``harbor.plugins`` entry-point group; Arctic-side code is a dependency, not the entry point. Package moves out of ``arctic_platform/integrations/harbor/`` into a top-level ``harbor_cortex_backend/`` package with its own ``pyproject.toml``: arctic_platform/integrations/harbor/cortex_agent.py -> harbor_cortex_backend/agent.py arctic_platform/integrations/harbor/host_environment.py -> harbor_cortex_backend/env.py arctic_platform/integrations/harbor/harbor_runner.py -> harbor_cortex_backend/train.py arctic_platform/integrations/harbor/aggregate_runs.py -> harbor_cortex_backend/aggregate.py (and models.py, backend.py, adapter.py, task_gen.py move unchanged) pyproject.toml registers two entries under ``harbor.plugins``: arctic-cortex-agent -> harbor_cortex_backend.agent:CortexRLAgent arctic-cortex-env -> harbor_cortex_backend.env:HostEnvironment so ``harbor plugins list`` shows them, and ``harbor run --agent arctic-cortex-agent --env arctic-cortex-env ...`` resolves them through Harbor's own resolver. Two console scripts are installed with the wheel: harbor-cortex-train the ``train.py`` driver (was ``python -m arctic_platform.integrations.harbor.harbor_runner``) harbor-cortex-aggregate multi-seed CI aggregator Verified end-to-end after ``pip install -e ./harbor_cortex_backend``: * ``harbor plugins list`` shows both entries * ``resolve_plugin_import_path('arctic-cortex-agent')`` returns our class * ``import_class(...)`` confirms ``CortexRLAgent`` is a Harbor ``BaseAgent`` and ``HostEnvironment`` is a ``BaseEnvironment`` * Harbor's stock ``Verifier`` + ``HostEnvironment`` + our ``test.sh`` scoring smoke test still passes (correct/close-wrong/no-int) README rewritten around the Harbor-user's install flow (``pip install harbor harbor-cortex-backend`` then ``harbor plugins list``), with the train + eval commands using short names. HANDOFF.md updated to reflect the reversed direction. Co-authored-by: Cursor <cursoragent@cursor.com>

…pe as verl) Follow-up to the plugin reposition. The previous change put the code in a brand-new top-level ``harbor_cortex_backend/`` package with its own ``pyproject.toml``. Convention in this repo is that framework glue lives under ``arctic_platform/integrations/<framework>/`` (see the sibling ``arctic_platform/integrations/verl/`` adapter), so this commit moves the harbor plugin there. Layout: arctic_platform/integrations/ verl/ # unchanged harbor/ # new: was harbor_cortex_backend/ __init__.py, agent.py, env.py, backend.py, models.py, adapter.py, task_gen.py, train.py, aggregate.py, README.md, RUN_LOG.md (no nested pyproject.toml) The nested ``pyproject.toml`` is dropped; the two ``harbor.plugins`` entry points and the two ``harbor-cortex-*`` console scripts are now registered on Arctic-Platform's top-level ``pyproject.toml``: [project.entry-points."harbor.plugins"] arctic-cortex-agent = "arctic_platform.integrations.harbor.agent:CortexRLAgent" arctic-cortex-env = "arctic_platform.integrations.harbor.env:HostEnvironment" [project.scripts] harbor-cortex-train = "arctic_platform.integrations.harbor.train:cli" harbor-cortex-aggregate = "arctic_platform.integrations.harbor.aggregate:main" A ``[project.optional-dependencies].harbor`` extra is added so users install with ``pip install 'arctic_platform[harbor]'``. All imports and default agent/env paths inside the plugin are rewritten from ``harbor_cortex_backend.*`` back to ``arctic_platform.integrations.harbor.*``. README + HANDOFF updated. Verified after reinstall: * ``harbor plugins list`` shows both entries under the new paths * ``harbor-cortex-train --help`` runs * Full E2E: 3-step GRPO run on Cortex QA6 completed in 6m — every ``harbor run`` subprocess resolved and ran ``CortexRLAgent`` + ``HostEnvironment`` from ``arctic_platform.integrations.harbor.*`` Co-authored-by: Cursor <cursoragent@cursor.com>

Drops the personal absolute path ``/home/yak/miniconda3/envs/skyrl_arl/bin/harbor`` from the captured console transcript (17 sites) — the log now just shows ``harbor run …`` as any reader would invoke it. Updates the transcript's import-path references from the pre-rename ``arctic_platform.integrations.harbor.cortex_agent:CortexRLAgent`` / ``.host_environment:HostEnvironment`` to the current ``arctic_platform.integrations.harbor.agent:CortexRLAgent`` / ``.env:HostEnvironment`` so the log matches what the code actually is today. No changes to result numbers, Cortex sub-job UUIDs (which are already invalidated), timestamps, or any other transcript content. Co-authored-by: Cursor <cursoragent@cursor.com>

Self-review pass. No behavior changes. Removes: * Repetitive "real"/"stock"/"clean" adjectives and defensive bolding ("Every LLM call runs inside a **real** ``harbor run`` trial (real ``BaseAgent``, real ``BaseEnvironment``, real ``RolloutDetail``)" -> state it once, in plain prose). * The "What it proves" bullet list on the README — the same points are already carried by the "how it works" paragraph and the file table. * "That's the whole invocation." / "Nothing in this list is Arctic-specific plumbing" / "— none of our code changes." — filler / bragging. * Broken relative link to the RFC (RFC lives outside the Arctic-Platform repo; reference it by name instead of by path). * Snowflake-specific example ("locked-down Snowflake VM") from env.py's docstring — generalized to "hosts without Docker / podman / user-namespace access". * Bolded / marketing phrasing around the +10 pp mean-reward result; the tight CI already carries the message. Co-authored-by: Cursor <cursoragent@cursor.com>

Two changes on the plugin's data plane so any Harbor agent — not just our reference CortexRLAgent — can drive training on Cortex: * adapter: flatten Harbor's per-turn RolloutDetail (list[list[int]]) into (prompt, completion, loss_mask). loss_mask marks every model-produced token across all turns as trainable, so multi-turn agents (Terminus 2, most CLI-shelling agents) get gradient signal on intermediate reasoning instead of only turn 0. Falls back to final-turn-only when Harbor's prompt[i+1] == prompt[i]+completion[i] invariant doesn't hold. * train.py: --sampling-api-base + --llm-backend flags. When set, the runner points harbor run at the sub-job's OpenAI-compat URL via --model-base-url + --ak api_base= (both stock Harbor flags) instead of the reconnect-config path, and no longer forces our CortexRLAgent. The hop is a no-op until Cortex exposes /v1/chat/completions on the sampling sub-job (tracked in PLAN.md). Also: 6 unit tests for _flatten_multi_turn (single-turn, 2-turn / 3-turn invariant holds, invariant break, empty inputs, load_job_dir populates loss_mask); make __init__.py lazy so tests can import adapter/models without harbor installed. Co-authored-by: Cursor <cursoragent@cursor.com>

Documents where the integration is (native mode, E2E on Cortex QA6) and what remains for "any Harbor agent trains on Cortex": one Cortex-side change to proxy vLLM's OpenAI routes on the sampling sub-job, plus two already-drafted Harbor upstream ergonomic patches. Neither is required for the current PR to be useful. README grows a "Two sampling modes" table and a worked example of the OpenAI-compat invocation (`--sampling-api-base ... --agent terminus_2`), and the Follow-ups section now points at PLAN.md. Co-authored-by: Cursor <cursoragent@cursor.com>

- Drop internal deployment name ("QA6") from PLAN.md, README.md, RUN_LOG.md. - Drop internal branch reference from PLAN.md ("arctic/plugin-..." branch name; the companion RFC path is enough). - Trim recent docstrings in models.py / adapter.py / train.py: drop repeated "so gradient flows through intermediate reasoning" / "not just the final action" phrasing, keep the mechanical description. - Trim README's "Two sampling modes" section (drop marketing framing, keep the table + example). - Shorten the __init__.py lazy-import note. No behavior changes; tests unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>

Adds /v1/chat/completions, /v1/completions, /v1/models to the same FastAPI server that hosts /generate. Any OpenAI-SDK client (Terminus 2 via LiteLLM, LangChain, the openai CLI) can now sample from a running sampling sub-job without RL-specific plumbing. Design: * No vLLM HTTP subprocess. The router in arctic_platform/openai_compat translates OpenAI requests directly into ReplicaPool.generate() calls, so scheduling / prefix cache / tensor-parallel routing continue to go through the existing pool. /generate and /v1/* share the same worker pool. * Lives at the top of arctic_platform/ (not under rl/) so tests can import the router without pulling in the training kernel (tensordict, Ray, DeepSpeed). * stream=True runs the full generation and replays it as SSE deltas in OpenAI's chunked shape. Client contract is correct; first-token latency is not. Incremental streaming needs a delta-yielding surface on ReplicaPool. * Chat template comes from the tokenizer loaded once at /initialize time (app.state.sampling_tokenizer). Models without a chat template return HTTP 400 pointing users at /v1/completions. * Unknown OpenAI fields (response_format, tools, ...) are accepted and ignored so new SDK versions don't 422. Testing: * 19 unit tests under tests/openai_compat/test_openai_compat.py drive the router with FastAPI's TestClient + a fake pool + fake tokenizer: parameter mapping, n>1, stop lists, streaming SSE structure, error paths (503 before init, 400 for no chat template, 400 for bad body), finish-reason mapping, unknown-field tolerance. * 4 real-SDK contract tests under tests/openai_compat/test_real_openai_sdk.py spin up a live uvicorn subprocess and drive it with the openai Python client: chat completions non-stream + stream, completions, models list. Skips cleanly when openai isn't installed. http_server.py wiring: * app.include_router(openai_compat_router). * /initialize (sampling branch) now loads AutoTokenizer for the model and stores it on app.state.sampling_tokenizer alongside sampling_pool and sampling_model_name. * /destroy clears the tokenizer + model_name on sampling teardown. * main() defaults the new state fields so /v1/* returns 503 (models returns empty) before any sampling job initializes. Co-authored-by: Cursor <cursoragent@cursor.com>

* train.py: --sampling-api-base auto derives the URL from the connected client's transport. On-prem the client's transport carries a base_url (http://host:port); we append /v1 to hit the OpenAI-compat surface added in the previous commit. Cortex raises a clear message pointing users at PLAN.md until sub-job URL routing lands there. * README: "Sampling modes" table now describes both paths as working today on-prem, and shows the two-step demo (start the RL server, drive with harbor-cortex-train --agent terminus_2 --sampling-api-base auto) that trains any stock Harbor agent without touching Harbor. * PLAN.md: rewritten to reflect that the OpenAI-compat routes ship in this PR, not "blocked on Cortex". The only Cortex-side item left is sub-job URL routing at the ingress, which is control-plane work outside this repo. Co-authored-by: Cursor <cursoragent@cursor.com>

…e works Harbor's LiteLLM backend (harbor.llms.lite_llm.LiteLLM._extract_token_ids) populates RolloutDetail token ids from vLLM's OpenAI-server extensions: top-level `prompt_token_ids` and per-choice `token_ids`. Our router was returning only the standard OpenAI shape, so any Harbor agent sampling through /v1/chat/completions with collect_rollout_details=True would get an empty RolloutDetail and the adapter would silently drop the trial. * Emit `prompt_token_ids` (tokenized rendered prompt) at the top level. * Emit `token_ids` per choice (from the pool result) on non-stream and the terminal SSE frame. * Add integration test that runs a live uvicorn + real `harbor.llms.lite_llm.LiteLLM(collect_rollout_details=True)` and asserts both fields come back non-empty. Closes the client-side E2E loop for any Harbor agent that speaks OpenAI-chat via LiteLLM, without needing GPUs to prove the wiring. * PLAN.md updated with honest E2E status: router path is proven through the real Harbor pipe; GPU-backed loop with real vLLM + Terminus 2 is still to be run on a GPU host. Co-authored-by: Cursor <cursoragent@cursor.com>

The only thing this branch actually observed converge is native mode (CortexRLAgent -> Cortex) on Qwen3-0.6B, per RUN_LOG.md. Everything about the OpenAI-compat "any Harbor agent" path is code + wire-shape tests against a fake ReplicaPool + stub tokenizer -- necessary but not sufficient. Nothing here has been booted against a real vLLM or run through a real Terminus 2 training loop. * README.md: drop "both work today on-prem"; mark OpenAI-compat mode as implemented but not yet exercised. * PLAN.md: rewrite the status table; add an explicit "Not yet real" section that lists what a genuine E2E requires (arctic_platform.rl.http_server with real GPUs + real Qwen tokenizer + Terminus 2 loop). * test_real_openai_sdk.py, test_harbor_litellm_integration.py: retitle as wire-shape tests. They still exercise the real openai/LiteLLM clients against our router, but the pool + tokenizer are fake -- the docstrings now say so. Co-authored-by: Cursor <cursoragent@cursor.com>

…of work Co-authored-by: Cursor <cursoragent@cursor.com>

…3-1.7B Turns the Harbor plugin from "arithmetic reference only" into "any OpenAI-compat Harbor agent, any Cortex-supported model, any Harbor task pack", validated end-to-end on a real verifiable-reward benchmark. Gateway path ------------ - ``arctic_platform.integrations.harbor.openai_gateway.DriverOpenAIGateway``: driver-local FastAPI + uvicorn on ``127.0.0.1`` that re-uses the existing ``arctic_platform.openai_compat`` router and forwards every ``/v1/chat/completions`` call to ``ArcticRLClient.generate`` over the Cortex ``operation`` envelope. No Cortex control-plane change, no monkey-patch on Harbor. - ``arctic_platform.integrations.harbor.litellm_chat_agent.LiteLLMChatAgent``: minimum-viable Harbor ``BaseAgent`` using ``harbor.llms.lite_llm.LiteLLM``. Captures ``prompt_token_ids``/``completion_token_ids`` from the gateway (vLLM OpenAI extension) into ``RolloutDetail`` for GRPO; catches ``OutputLengthExceededError`` so truncated rollouts still score. - ``openai_compat._render_chat_prompt`` disables Qwen3's ``enable_thinking`` in the chat template by default; ``extra_body.chat_template_kwargs`` overrides per request. Verbose ``<think>...</think>`` reasoning was eating the ``max_tokens`` budget before the answer materialised. - ``train._derive_openai_base_url`` boots + returns the gateway when ``client.transport`` has no ``base_url`` (Cortex path); shuts it down in the same ``finally`` block that tears the Cortex job down. Harbor task-pack compatibility ------------------------------ - ``_write_step_manifest`` now symlinks chosen task dirs into the step dataset root. Harbor's ``harbor run -p <dir>`` walks ``path.iterdir()`` — it doesn't parse ``dataset.toml`` — so the previous manifest form failed with ``ValueError: Either datasets or tasks must be provided.`` on step 0. - ``HostEnvironment.upload_dir`` / ``upload_file`` now mirror ``_rewrite_paths`` on the *contents* of uploaded shell/python/txt files (``_REWRITE_SUFFIXES`` allowlist keeps binaries untouched). Task shell scripts hard-code ``python3 /tests/test_output.py``, which under Docker resolves to the container's real filesystem but on ``HostEnvironment`` failed with ``can't open file '/tests/test_output.py': [Errno 2] No such file...`` - ``/workspace`` added to ``_ROOTED_PREFIXES`` (reasoning-gym / open-thought convention). - ``train.py`` gains ``--train-gpus`` / ``--sample-gpus`` / ``--max-seq-len`` for scaling beyond the 0.6B reference; defaults match PR #66's arithmetic recipe. E2E validation -------------- Reproduced the arithmetic reference through the gateway (matches native mode: pass@1 +0.159 vs. native +0.029 mean of 3 seeds), then ran Qwen3-1.7B on reasoning-gym-easy (72 train / 24 held-out, 8 GRPO steps, 1T+1S GPU) end-to-end: BASELINE pass@1 = 0.043 (1/23) mean reward 0.088 FINAL pass@1 = 0.087 (2/23) mean reward 0.119 RESULT Δ pass@1 +0.043 Δ mean reward +0.031 Both metrics improve monotonically; noisy per-step reward is expected for a mixed-family reasoning benchmark on a 1.7B model. Full run identifiers, headline table, and reproduce block in RUN_LOG.md. Qwen3-4B: not currently supported by the Cortex-training QA6 image (sub_job_failed inside 2min on both 1x1 and 2x2 GPU shapes with no error text through the SnowAPI status endpoint). 1.7B is the current ceiling on Cortex. Tests: ``tests/openai_compat/test_driver_gateway.py`` boots the gateway with a fake ``ArcticRLClient`` + real Qwen tokenizer and exercises ``/v1/models``, ``/v1/chat/completions`` (incl. ``n>1`` batching), and the full ``harbor.llms.lite_llm.LiteLLM`` integration including ``collect_rollout_details=True`` and vLLM token-id extraction. Co-authored-by: Cursor <cursoragent@cursor.com>

sfc-gh-mwyatt and others added 22 commits August 5, 2026 15:23

cortex transport

375684a

Co-authored-by: Cursor <cursoragent@cursor.com>

nested backend config models

c6cd017

update configs to capture more options, better organize fields/nesting

08dd0e1

shim: fix loss_mask fallback to use is-None checks (bool(tensor) rais…

af05514

…es on multi-element) Co-authored-by: Cursor <cursoragent@cursor.com>

harbor: CHECKPOINT.md — pause-and-resume handoff doc for this thread …

97c1f7b

…of work Co-authored-by: Cursor <cursoragent@cursor.com>

sfc-gh-kganesan force-pushed the sfc-gh-kganesan/skyrl-cortex-shim branch 2 times, most recently from ad948b0 to 489a238 Compare August 13, 2026 22:09

sfc-gh-kganesan force-pushed the sfc-gh-kganesan/skyrl-cortex-shim branch 3 times, most recently from 60ba6bf to 37dce1e Compare August 26, 2026 04:06

sfc-gh-kganesan force-pushed the sfc-gh-kganesan/skyrl-cortex-shim branch 2 times, most recently from 60ba6bf to 4b44583 Compare August 27, 2026 22:05

Navigation Menu

Uh oh!

harbor: Cortex training backend as a Harbor plugin (draft) - #66

harbor: Cortex training backend as a Harbor plugin (draft)#66
sfc-gh-kganesan wants to merge 23 commits into
sfc-gh-kganesan/skyrl-cortex-shimfrom
sfc-gh-kganesan/harbor-cortex-backend

sfc-gh-kganesan commented Aug 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sfc-gh-kganesan commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Two sampling modes (both working E2E)

End-to-end demo: Terminus 2 training on Cortex, on-prem

What's in the package

arctic_platform/openai_compat.py (new; 605 LOC)

arctic_platform/rl/http_server.py

Harbor plugin (arctic_platform/integrations/harbor/)

Tests

Experiment: 15-step GRPO on Cortex (native mode)

Non-obvious client-side bugs found and fixed during bringup

What's still outside this repo

Sanitized for a public PR

Companion RFC

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sfc-gh-kganesan commented Aug 10, 2026 •

edited

Loading

`arctic_platform/openai_compat.py` (new; 605 LOC)

`arctic_platform/rl/http_server.py`

Harbor plugin (`arctic_platform/integrations/harbor/`)