Description:
When fine-grained activation offloading is combined with TransformerEngine partial CUDA graphs on a MoE model (DeepSeek V3 lite), training crashes on the first forward right after CUDA graph capture:
RuntimeError: Trying to resize storage that is not resizable
Location: megatron/core/pipeline_parallel/fine_grained_activation_offload.py, bulk_offload():
release_tensor.record_stream(cur_stream)
release_tensor.untyped_storage().resize_(0) # throws here
Environment:
Megatron-LM: Release0.19
TransformerEngine: 2.14.1
torch: 2.10
Hardware: A100*8
Reproduction:
--cuda-graph-impl transformer_engine
--cuda-graph-scope moe_router moe_preprocess attn
--optimizer-cuda-graph
--fine-grained-activation-offloading
--offload-modules core_attn attn_proj expert_fc1
Fails on the first training step, immediately after "Start CUDA Graphs capture... / Time spent in CUDA Graphs capture".
Stack trace (rank0)
experts.py:825 fc1_output = expert_fc1_manager.group_offload(
fc1_output, forced_released_tensors=[permuted_local_hidden_states], ...)
fine_grained_activation_offload.py:1288 fine_grained_offloading_group_offload(...)
fine_grained_activation_offload.py:1235 FineGrainedOffloadingGroupCommitFunction.forward
fine_grained_activation_offload.py:1129 on_group_commit_forward(name, forced_released_tensors)
fine_grained_activation_offload.py:1106 release_tensor.untyped_storage().resize_(0)
RuntimeError: Trying to resize storage that is not resizable
Description:
When fine-grained activation offloading is combined with TransformerEngine partial CUDA graphs on a MoE model (DeepSeek V3 lite), training crashes on the first forward right after CUDA graph capture:
RuntimeError: Trying to resize storage that is not resizable
Location: megatron/core/pipeline_parallel/fine_grained_activation_offload.py, bulk_offload():
release_tensor.record_stream(cur_stream)
release_tensor.untyped_storage().resize_(0) # throws here
Environment:
Megatron-LM: Release0.19
TransformerEngine: 2.14.1
torch: 2.10
Hardware: A100*8
Reproduction:
--cuda-graph-impl transformer_engine
--cuda-graph-scope moe_router moe_preprocess attn
--optimizer-cuda-graph
--fine-grained-activation-offloading
--offload-modules core_attn attn_proj expert_fc1
Fails on the first training step, immediately after "Start CUDA Graphs capture... / Time spent in CUDA Graphs capture".
Stack trace (rank0)
experts.py:825 fc1_output = expert_fc1_manager.group_offload(
fc1_output, forced_released_tensors=[permuted_local_hidden_states], ...)
fine_grained_activation_offload.py:1288 fine_grained_offloading_group_offload(...)
fine_grained_activation_offload.py:1235 FineGrainedOffloadingGroupCommitFunction.forward
fine_grained_activation_offload.py:1129 on_group_commit_forward(name, forced_released_tensors)
fine_grained_activation_offload.py:1106 release_tensor.untyped_storage().resize_(0)
RuntimeError: Trying to resize storage that is not resizable