actor rollout hidden states Transferqueue implement - #48
Open
zjuKirito wants to merge 5 commits into
Open
Conversation
…idden] 密集形式（用 unbind+cat
Conflict was in _resolve_hidden_state_chunks call site: - transferqueue branch added cache=self._tq_chunk_cache (cross-sample get cache) - upstream/main reformatted to multi-line (pre-commit/black) Resolution: keep cache param + adopt upstream multi-line formatting Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TQ_experiment_report.md
SpeCo 自有的 hidden state 传输已全部覆盖。剩余的标准 PPO 数据面是 verl 上游 TQ 的范畴,需切
main_ppo_sync,属更大工程,且仅多机有实质收益。单机场景下 SpeCo 的 TQ 能做的已完成。具体实验参考数据参考markdown，和基线对比实验，数据无损(acceptance_length 匹配)、无报错、drafter 训练正常。在timing_s/old_log_prob ，用tq实现hidden state，每两步收集一次，平均劣化3秒，端到端偏差较大，原因是波动本身较大（跑两次基线实验验证，端到端波动较大）。
劣化解释：
hidden state本就用
ray.putray.get绕过 driver,TQ 只是换了传输方式(且 GET 更慢)。多机有(跨节点 DataProto 传输),单机有限(本地 driver 无瓶颈)transfer_queue_verify_20260802_030633_eurus_rloo_drafter_online_0801_czy_trasnferqueue.log
transfer_queue_verify_20260802_074255_eurus_rloo_drafter_online_no_tq_0801_main.log