[draft] proposal: generic trainer.backend registry (alternative to PR #1)

[sfc-gh-kganesan]

Summary

Alternative refactor for the Arctic RL integration in response to PR #1 review. Honors the reviewer's "core stays integration-agnostic" principle while preserving the property that any existing SkyRL recipe can flip to Arctic via a single flag — without forking the recipe or changing its entrypoint.

This PR is a draft for discussion, parallel to #1. No reviewers tagged.

Reasoning — why this differs from the harbor pattern

	Environment plug-in (harbor, modal, openenv, …)	Training backend (Arctic RL, future Megatron, NeMo, …)
What it replaces	Reward source / dataset / env loop	DeepSpeed engine + inference engine + worker actors
Affects core trainer?	No	Yes — RayPPOTrainer → ArcticPPOTrainer
Affects core generator?	No	Yes — Generator → ArcticGenerator
Per-recipe portability	Each env is a different experiment	Should swap underneath any recipe
User mental model	"I want this env"	"I want faster training on whatever I'm doing"

The harbor pattern fits envs naturally because env = experiment. For training backends, users expect orthogonal swapping — analogous to trainer.algorithm.advantage_estimator=grpo|reinforce|.... Strict harbor pattern would require forking every standard SkyRL recipe (run_gsm8k.sh, run_math.sh, etc.) into integrations/arctic-rl/examples/ to enable Arctic RL.

This PR proposes a generic backend registry: one trainer.backend: str field on core TrainerConfig plus a 3-line lazy dispatch in main_base.main() that imports f"{backend}.entrypoint". No integration name is hardcoded in core.

Design

USER
  python -m skyrl.train.entrypoints.main_base \
      trainer.backend=arctic_rl trainer.arctic_rl={} <flags>
  OR (harbor-style)
  uv run --extra arctic-rl -m arctic_rl.entrypoint <flags>
                         │
                         ▼
skyrl/train/entrypoints/main_base.py:main()
  1. Peek `trainer.backend=` from raw sys.argv
  2. if backend != "fsdp":
       backend_main = import_module(f"{backend}.entrypoint").main
       return backend_main()             ◄── 3 lines, generic
  3. Else: parse SkyRLTrainConfig.from_cli_overrides (FSDP path)
                         │
        ┌────────────────┴────────────────┐
        │ backend = fsdp                  │ backend = arctic_rl  (or megatron, nemo, ...)
        ▼                                 ▼
Standard FSDP path                arctic_rl.entrypoint:main()
  RayPPOTrainer + Generator         Parse with ArcticSkyRLConfig = make_config(trainer_cls=ArcticTrainerConfig)
                                    Forward ARCTIC_* env vars to Ray
                                    ArcticPPOTrainer + ArcticGenerator (DeepSpeed + ArcticInference vLLM)

Code layout

skyrl-public/
├── skyrl/train/                                  ← CORE: integration-agnostic
│   ├── config/config.py
│   │     TrainerConfig.backend: str = "fsdp"     ← only addition (generic field)
│   ├── entrypoints/main_base.py
│   │     def main():
│   │         # peek sys.argv → dispatch          ← only addition (3 lines)
│   └── utils/utils.py                            ← unchanged from upstream
│
├── pyproject.toml
│     [project.optional-dependencies]
│     arctic-rl = ["skyrl[skyrl-train]", "arctic_training"]   ← extras group
│     [tool.setuptools.packages.find]
│     include = ["skyrl*"]                                     ← reverted to upstream
│
└── integrations/arctic-rl/                       ← INTEGRATION: self-contained
    ├── arctic_rl/
    │   ├── config.py
    │   │     ArcticRLTrainerConfig (MOVED FROM CORE)
    │   │     ArcticTrainerConfig(TrainerConfig) with arctic_rl field
    │   │     ArcticSkyRLConfig = make_config(trainer_cls=ArcticTrainerConfig)
    │   │     build_rl_config(cfg) → ArcticRLClientConfig
    │   ├── entrypoint.py     (main() parses with ArcticSkyRLConfig + inlined env-var forwarding)
    │   ├── trainer.py        (ArcticPPOTrainer)
    │   └── generator.py      (ArcticGenerator)
    └── examples/run_gsm8k_grpo_4gpu.sh

Reviewer's 5 asks — all addressed

Reviewer feedback	Status
Revert [tool.setuptools.packages.find] to upstream	✅ include = ["skyrl*"]
Move ArcticRLTrainerConfig out of core config.py	✅
Remove _propagate_arctic_env_vars from core utils	✅ inlined in arctic's entrypoint
Remove arctic-specific routing from main_base.py	✅ replaced with generic dispatch (no arctic_rl strings in core code)
Use make_config(trainer_cls=...) extension	✅ ArcticSkyRLConfig = make_config(trainer_cls=ArcticTrainerConfig)
uv run --extra arctic-rl -m ...entrypoint mode supported	✅ AND trainer.backend=arctic_rl flag mode also works

One delta from harbor

Core gains exactly:

• 1 generic field: backend: str = "fsdp" on TrainerConfig
• 3 dispatch lines in main_base.main() (peek argv → import_module(f"{backend}.entrypoint").main)

Both integration-agnostic. Same architectural lever as make_config(trainer_cls=...) (the reviewer-blessed extension path) but at the entrypoint level.

Convergence test ✅

Validated end-to-end with the original arctic-rl-public-sf-validated stack (1×2 H200 colocated, GSM8K GRPO, Qwen3-0.6B), via the new trainer.backend=arctic_rl dispatch. 9 GRPO steps, reward climbing:

Step	avg_pass@4	avg_raw_reward	sec/it
1	0.250	0.125	67.3s
2	0.250	0.063	34.8s
3	0.250	0.250	23.9s
4	0.500	0.375	18.8s
5	0.750	0.625	14.9s
6	0.250	0.125	13.6s
7	1.000	0.500	12.9s
8	0.250	0.063	12.6s
9	(running)	(running)	12.1s

Same noisy-but-improving GRPO shape as PR #1's validation table. Steady-state ~12s/iter. trainer/global_step: 9 confirmed.

Diff summary

7 files, +107/-86 lines. Core net change: -60 lines (mostly ArcticRLTrainerConfig moving out). Integration: +60 (the moved class + harbor-style extension).

Test plan

• python -m skyrl.train.entrypoints.main_base trainer.backend=arctic_rl trainer.arctic_rl={} ... reaches arctic flow
• Convergence over 9 GRPO steps with reward climbing
• No core file imports arctic_rl or arctic_training (grep -rn arctic skyrl/ returns only docstring examples)
• uv run --extra arctic-rl -m arctic_rl.entrypoint ... (harbor-style direct mode) — not yet tested but trivially equivalent
• FSDP recipes (no arctic flags) still work after this PR — should be no-op since backend defaults to "fsdp"