Partial Rollout #350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

yytang0204 merged 22 commits into alibaba:main from yytang0204:support_partial_rollout

Sep 4, 2025

Collaborator

yytang0204 commented Aug 25, 2025

Support partial rollout

yytang0204 added 11 commits

August 25, 2025 11:23


          [WIP] Partial Rollout

68590ba


          Support config for rollout

6869ff1


          merge main

bc8506d


          support dynamic rule reward

649bf58


          support sglang for partial rollout; support megatron for dynamic trai…

5a436c0

…n batch


          remove redundant files

61bd84f


          cleanup code

2d4fa53


          fix pylint

247858f


          fix bug

2c041fb


          fix pylint; fix sglang

95b91ea


          fix pylint

2e3ce5e

yytang0204 marked this pull request as ready for review

September 2, 2025 05:37

yytang0204 changed the title ~~[WIP] Partial Rollout~~ Partial Rollout

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo.py Show resolved Hide resolved

haiasd reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/vllm_policy_inference.py Outdated Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/utils/parse_utils.py Outdated Show resolved Hide resolved

yytang0204 added 4 commits

September 3, 2025 17:43


          fix adv calculation

a33b8d7


          fix pylint

d2ddb7e


          fix pylint

3a2b03e


          fix redundant import

ecb1c7e

haiasd reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Outdated Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/advantage_compute.py Outdated Show resolved Hide resolved

haiasd reviewed

View reviewed changes

scripts/train_fsdp_vllm_qwen3_8b_grpo_local_partial.sh Outdated Show resolved Hide resolved

haiasd reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Show resolved Hide resolved

haiasd reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Outdated Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/advantage_compute.py Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Show resolved Hide resolved

haiasd reviewed

View reviewed changes

template/grpo_megatron.yaml Show resolved Hide resolved

template/grpo_megatron.yaml Outdated Show resolved Hide resolved

haiasd reviewed

View reviewed changes

chatlearn/data/prompt_dataset.py Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Outdated Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Outdated Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Outdated Show resolved Hide resolved

haiasd reviewed

View reviewed changes

chatlearn/models/sglang_module.py Show resolved Hide resolved

yytang0204 added 4 commits

September 4, 2025 11:16


          remove bash

caac8d0


          merge main for cp and vl

8b769a2


          fix comments

50391d9


          fix pylint

4856ab6

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/megatron_policy_trainer.py Outdated Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

chatlearn/algorithm/grpo_utils/rollout_manager.py Show resolved Hide resolved

lostkevin reviewed

View reviewed changes

scripts/train_mcore_vllm_qwen3_8b_grpo_partial_rollout.sh Outdated Show resolved Hide resolved


          fix comments; fix pylint

806a93c

lostkevin previously approved these changes

View reviewed changes

Collaborator

lostkevin left a comment

LGTM


          Add explain for data dict

d739788

yytang0204 dismissed lostkevin’s stale review via

d739788

September 4, 2025 06:56


          Add explain for data dict

ab5e9b5

haiasd approved these changes

View reviewed changes

lostkevin approved these changes

View reviewed changes

yytang0204 merged commit 4f7051f into alibaba:main

3 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet