Skip to content

RuntimeError: Trying to resize storage that is not resizable in fine-grained activation offloading when force-releasing MoE expert_fc1 input that is a CUDA-graph static output #7009

Description

@RayXL97

Description:
When fine-grained activation offloading is combined with TransformerEngine partial CUDA graphs on a MoE model (DeepSeek V3 lite), training crashes on the first forward right after CUDA graph capture:
RuntimeError: Trying to resize storage that is not resizable

Location: megatron/core/pipeline_parallel/fine_grained_activation_offload.py, bulk_offload():
release_tensor.record_stream(cur_stream)
release_tensor.untyped_storage().resize_(0) # throws here

Environment:
Megatron-LM: Release0.19
TransformerEngine: 2.14.1
torch: 2.10
Hardware: A100*8

Reproduction:
--cuda-graph-impl transformer_engine
--cuda-graph-scope moe_router moe_preprocess attn
--optimizer-cuda-graph
--fine-grained-activation-offloading
--offload-modules core_attn attn_proj expert_fc1

Fails on the first training step, immediately after "Start CUDA Graphs capture... / Time spent in CUDA Graphs capture".

Stack trace (rank0)
experts.py:825 fc1_output = expert_fc1_manager.group_offload(
fc1_output, forced_released_tensors=[permuted_local_hidden_states], ...)
fine_grained_activation_offload.py:1288 fine_grained_offloading_group_offload(...)
fine_grained_activation_offload.py:1235 FineGrainedOffloadingGroupCommitFunction.forward
fine_grained_activation_offload.py:1129 on_group_commit_forward(name, forced_released_tensors)
fine_grained_activation_offload.py:1106 release_tensor.untyped_storage().resize_(0)
RuntimeError: Trying to resize storage that is not resizable

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions