Skip to content

txt2sql (BIRD) single-node FSDP baseline crashes silently at Ray startup #38

Description

@sfc-gh-kganesan

Summary

The single-node FSDP-native baseline for the txt2sql (BIRD) recipe —
recipes/rl/skyrl/txt2sql/run_qwen3_8b_bird_grpo_fsdp.sh — crashes silently
during Ray startup (worker dies ~30s after ray init, no traceback propagated
to the driver). The Arctic RL sibling (run_qwen3_8b_bird_grpo_arl.sh) comes up
and trains fine on the same box, and the LoongRL FSDP baseline
(long_context_qa/run_qwen3_8b_loongrl_grpo_fsdp.sh) also works — so this is
specific to the BIRD + FSDP path on a single node.

Impact

Blocks the ZoRRo speedup A/B for txt2sql on a single node. Convergence and
the ARL path for BIRD are validated; only the FSDP wall-clock baseline is
missing, so the BIRD speedup number is not yet reproduced end-to-end.

What was tried

  • Cleared orphaned Ray processes + leaked /dev/shm IPC segments.
  • Confirmed PromptDataset builds correctly in isolation (fork and spawn).
  • Ran BasePPOExp inline (no Ray remote) — still a silent exit.
  • Combed Ray session logs (worker-*.err/out, raylet.out, monitor.log) —
    no traceback; Ray reports the remote task as never actually started.

Likely area

Worker-side setup (placement group / vLLM engine init) for the BIRD FSDP config
failing without error propagation; possibly environment instability after
repeated force-kills. Needs a clean-box repro with Ray error propagation forced
on.

Acceptance

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions