Megatron-LM Monthly Activity Report — August 2026 (Final, through August 31, 2026)

[lhb8125]

Megatron-LM Monthly Activity Report — August 2026 (Final, through August 31, 2026)

Chinese version: 中文版本

Reporting window: 2026-08-01 through 2026-08-31 (Asia/Shanghai); source: NVIDIA/Megatron-LM.

Overview

Delivered work advanced MFSDP and parallel training, CUDA Graph execution, hybrid architectures, attention kernels, inference serving, RL durability, and model refitting. Ongoing work advances distributed training, model architecture, MoE and inference execution, checkpointing, optimizer strategies, and runtime reliability.

• Substantive PR activity after filtering: opened 667; merged 334; closed without merge 208; active at cutoff 349.
• Excluded from the report and statistics: test-only 42, CI-only 44, test/CI-only 0, format-only 0.

Delivered

MFSDP sharding, gradients, and optimizer memory

MFSDP gained broader sharding and optimizer capabilities with stronger gradient, tied-parameter, communication-buffer, and weight-synchronization correctness.

• MFSDP v2 now scales expert gradients correctly, handles recomputation phases, preserves tied-parameter aliases, allocates persistent gradients on the reduce-scatter stream, refreshes BF16 compute weights after optimizer steps, and validates fused MLA/MTP checkpoint naming for unfused-to-fused and fused round-trip loading. (#5956, #6153, #6164, #6187, #6271, #6336)
• FSDP communication supports configurable persistent buffers, while optimizer-state and selected master-weight updates can run in bounded GPU memory backed by pinned CPU storage. (#6244, #6316)

CUDA Graph recomputation and overlap

CUDA Graph paths gained safer recomputation, capture-scoped communication, fused mHC execution, and hybrid-training correctness protections.

• Selective mHC recomputation now works under CUDA Graphs with EP all-to-all overlap, while Transformer Engine paged-stash captures manage capture order, auxiliary streams, runtime restoration, and capacity checks. (#5841, #6022, #6870)
• GTP backward reduce-scatter, EP A2A recomputation, and checkpoint-only parameter discovery now preserve capture-scoped ownership, activation lifetimes, and required autograd edges. (#6060, #6311, #6373, #6429)
• Hybrid models can use fused mHC execution and receive fixes for recomputation state, routing metadata, layer-norm checkpointing, and CUDA-Graph batch isolation. (#6401, #6704)
• GTP recomputation uses separate gather-buffer selection so asynchronous prefetch cannot overwrite weights still needed by a recomputed forward. (#6407)

GDP and hybrid model architectures

New and extended hybrid model paths support GDP, Ling-V3 Tiny, chunkwise context parallelism, and optimized hybrid layouts.

• Gated Delta Product became a first-class trainable and inferable hybrid component with TP/CP, packed sequences, dynamic inference, checkpoint resharding, and optimizer integration. (#6074)
• GDP adds activation recompute and offload controls, a CuTeDSL kernel path, and channel-last causal-convolution handling for training, prefill, and decode. (#6553)
• Training support was added for the Ling-V3 Tiny KDA–MLA hybrid sparse-MoE architecture, including gated MLA, MTP, QK clipping, and model-specific layer allocation. (#6556)
• GDP supports chunkwise context parallelism through distributed recurrent-state prefix and suffix combination, while hybrid models can keep GDN layers contiguous and convert layouts only around regular attention. (#6387, #6815)

Dynamic inference, KV-cache handoff, and multimodal serving

Inference serving expanded with resilient prefix caching, disaggregated state transfer, multimodal dynamic batching, and video support.

• Prefix-cache accounting now treats free and evictable blocks consistently for resumed requests, allowing pauses to resume when cached blocks can be evicted. (#5995)
• Prefill-to-decode handoff transfers KV cache through NIXL or NCCL, while hybrid handoff also transfers SSM convolution and recurrent states with resharding and partial prefix reuse. (#6222, #6584)
• Dynamic batching now supports image-aware VLM requests and video prompts, including preprocessing, placeholder expansion, vision execution, multimodal caching, and HTTP/API integration. (#6260, #6809)
• Batch-invariant Nemotron inference removes token-only padding around decode attention and SSM execution and restores it afterward to keep aligned CUDA-Graph batches from treating padding as requests. (#6893)

GTP, Muon, and distributed optimizer parallelism

Generalized tensor parallelism gained stronger gradient communication, readiness, memory, and Muon execution support.

• GTP supports repeated MTP-weight communication, with separate all-gathers and successor-specific reduce-scatter completion to preserve shared-weight gradients. (#6242)
• DP-balanced rerouting now uses padded per-field all-gathers within CP lanes, while GTP-sharded Muon balances shards by Newton–Schulz compute cost and supports blockwise, duplicated, and distributed orthogonalization. (#6378, #6379, #6380)
• DDP publishes parameter readiness before GTP prefetch, and GTP gains optional symmetric-memory registration and pooled weight-gradient buffers for dense and expert reduce-scatter. (#6388, #6535)

MLA, DSA, and Compressed Sparse Attention

Attention implementations gained fused positional processing, higher-precision DSA projections, a reference SBHD CSA path, and corrected fused loss normalization.

• TileLang DSA supports independent quantization and BF16/FP32 projection-output controls, while MLA and DSA RoPE packing uses fused Triton forward and backward paths. (#6279, #6343, #6842, #6846)
• CSA teacher and fused indexer-loss normalization now include compressed-key, sliding-window, and sink terms across SBHD, THD, packed, sparse, and dense paths. (#6349)
• A correctness-oriented native SBHD Compressed Sparse Attention implementation now covers compression, sparse and dense attention, sliding windows, sinks, and teacher LSE. (#6400)

Durable RL rollout storage

RL rollout processing gained persistence and recovery across process interruptions.

• Completed rollout groups are stored in a checksummed write-through rollout bank with restore, compaction, resume integration, and per-environment balancing. (#6352)

Batch-invariant MoE execution

Batch-invariant MoE training and inference gained selectable collective backends, gated-MoE execution, and aligned token-bucketing behavior.

• Batch-invariant MoE supports fused-vLLM, native Transformer Engine, and FlashInfer routed block-scale MXFP8 gated-MoE backends, with aligned CUDA-Graph token buckets, deterministic reduction behavior, and a bounded decode-only graph prefix with full-capacity fallback. (#6521, #6864)
• Selectable ordered and multimem reduction backends provide aligned token bucketing, GEMM choices, and exact-sum accumulation across supported collective paths. (#6806)

NCCL ReFIT and mutable MXFP8 weights

Refit workflows gained NCCL many-to-many transfer, bounded-memory MXFP8 updates, and communicator initialization fixes.

• NCCL many-to-many ReFIT supports topology caching, layout-agreement fingerprints, padding handling, and benchmark integration for transfers across parallel layouts. (#6712)
• MXFP8 inference weights can be updated in place with logical tensor metadata and refreshed derived caches, while large refits use parameter-aligned bounded batches to release staging and dequantization buffers between runs. (#6826, #6878)
• NCCLCopyService now eagerly connects communicators before checking queued work so idle ranks participate in resharding initialization and do not hang when only some ranks have copy operations. (#6955)

In Progress

MFSDP/FSDP scheduling and sharding

MFSDP and FSDP gain finer-grained lifecycle control, communication overlap, checkpoint support, and robust sharding behavior.

• Combined 1F1B and expert-parallel schedules coordinate MFSDP state transitions, prefetching, delayed reductions, and custom lifecycle hooks. (#6174, #6197, #6665, #6692, #6694)
• MFSDP v2 adds optimizer Distributed Checkpoint support, grouped-MoE storage remapping, hybrid sharding, and support for externally EP-sharded expert weights. (#6215, #6228, #6773, #6785)
• Deterministic user-buffer registration and per-wrapper NCCL pool ownership prevent rank-dependent layouts, resource aliasing, and registration failures. (#6317, #6923)

Checkpointing and distributed-state reliability

Checkpoint conversion, restoration, validation, RNG addressing, and asynchronous worker handling receive reliability improvements.

• CPU-only model conversion and checkpointing, quantized-parameter restoration, and optimizer-state slicing improve compatibility across device and precision transitions. (#6270, #6342, #6666)
• Uneven-DTensor preflight validation, data-parallel-aware RNG shard addressing, and model-owned process-group use improve distributed checkpoint consistency. (#6573, #6858, #6940)
• Asynchronous checkpoint worker failures now propagate rank-tagged errors, clear failed state, and permit worker replacement. (#6971)

MoE routing, expert execution, and inference kernels

MoE work expands routing policies and backend coverage while improving decode execution and expert-state handling.

• Dynamic MoE decode paths add optimized routing, GEMM, token-count, combine, attention, normalization, and diagnostic kernels, with expert-parallel graph initialization safeguards. (#6064, #6412)
• The MoE stack adds a pluggable MOK backend, dense top-k routing-map compatibility, batch-invariant combine behavior, and global-batch quantile balancing. (#6572, #6614, #6615, #6634, #6637, #6638)
• Expert duplication and Transformer Engine MXFP8 grouped-GEMM inference extend replica-aware placement, checkpoint handling, and device-resident routing. (#6892, #6933)
• Fused expert storage is organized as adjacent views so colocated generation can rebuild zero-copy serving views from updated weights. (#6963)

Context parallelism, THD, DSA, and MTP

Packed-sequence execution gains broader context-parallel layouts, MTP alignment, sparse-attention support, and state-passing paths.

• THD sequence-level MoE loss and context-parallel batch partitioning now derive ownership and shards from logical packed-sequence boundaries. (#6115, #6912)
• Context-parallel layouts become explicit per-batch and per-layer contracts, with runtime attention groups and latent MLA context parallelism supporting varied layouts and backends. (#6233, #6821, #6829)
• MTP gains layout-aware absolute sequence alignment for packed context-parallel inputs, while Mamba2 adds sequence-sharded state passing with halo and boundary-state exchange. (#6426, #6741, #6808)
• GQA-based DSA and its sparse backward paths handle minimum-memory execution and empty query rows with static safe tiles and zeroed gradients. (#6869, #6910, #6939)

DeepSeek, Qwen, and new model architectures

Model development adds heterogeneous decoder structures, hybrid attention, wide residual streams, and low-precision training paths.

• Qwen3.5-VL moves to HybridModel with GatedDeltaNet, attention, dense, and MoE layers, while generalized tensor parallelism spans training, checkpoint, inference, and MIMO paths; the Megatron Lite experimental stack adds QAT formats, router replay, dynamic context-parallel runtime support, streaming checkpoint conversion, and model integrations. (#6315, #6397, #6608)
• DeepSeek-V4 gains hybrid-layer construction, SBHD attention orchestration, fused CSA, hash routing, TP/CP attention paths, and learned indexer support. (#6402, #6403, #6404, #6405, #6887)
• Streamwise wide-residual support adds multiple-width residual streams, selective replay, ordinary-width MTP/MIMO boundaries, and shortcut-MoE integration. (#6716, #6805, #6855, #6959)
• Attention Residual aggregation, DeepSeek-V3 NVFP4 training, and SiTU-GLU Transformer Engine dispatch extend configurable model and precision behavior, while TE microbatch-state handling preserves correct accumulation behavior for reused layers after checkpoint resume. (#6673, #6674, #6840, #6841, #6965)

Muon and optimizer memory strategies

Muon gains new sharding and orthogonalization modes, while optimizer-state offloading reduces GPU residency.

• Per-head Muon orthogonalization and Hyperball constraints support QKV, MLA, GQA, gated, expert, and rematerialized layouts. (#6326, #6422)
• Layer-sharded and hybrid Muon strategies assign matrix updates across tensor-, expert-, and layer-sharding domains with separate dense and expert policies. (#6683, #6761)
• Chunked CPU offloading stores Adam moments and master weights outside GPU memory while avoiding full state rematerialization during checkpoint saves. (#6544)

Inference serving, rollout durability, and decoding

Inference and RL paths improve disaggregation, request lifecycle handling, selected-logprob computation, speculative replay, and durable rollout recovery.

• Disaggregated prefill/decode handoff transfers KV and recurrent SSM state between independently parallelized shards through coordinator-managed lifecycle operations. (#6675)
• Inference lifecycle fixes cover prefix-cache reconstruction, asynchronous request-state correctness, client-disconnect cancellation, and durable replay of incomplete rollout groups. (#6768, #6852, #6853, #6916)
• RL inference adds local-vocabulary selected-token logprob computation, while Mamba speculative decoding can replay state through checkpointed circular buffers. (#6617, #6706)

Deterministic runtime, CUDA Graphs, and extensibility

Runtime infrastructure improves deterministic execution, explicit process-group ownership, GTP graph support, metrics, and kernel backend selection.

• Deterministic Triton tuning and explicit process-group propagation extend across core model paths, process-group lifecycle management cleans up created communicators, and AsyncZMQ collective startup waits for subscriber readiness before the initial broadcast. (#6076, #6293, #6303, #6715, #6902)
• The centralized kernel backend registry and construction-time selection make optional implementations and incompatible configurations explicit, while configurable tensor metrics provide scheduled forward, parameter, and gradient observations with timing and TensorBoard, W&B, or stdout logging. (#6650, #6690)
• GTP adds persistent CUDA-graph buffers, symmetric-memory support, and consolidated sharding, communication, optimizer, and checkpoint integration. (#6713, #6867, #6882)
• Fused MLA RoPE synchronization and full-iteration CUDA Graph fixes address multi-warp races, capture behavior, grouped HybridModel layer discovery, and host-independent fixed-shape metadata construction for GatedDeltaNet THD capture. (#5963, #6796, #6958)

Scope and Method

The narrative is grouped by technical theme rather than PR chronology. Related dev and main PRs are described once. The rebuildable raw ledger, exclusion evidence, commit activity, and state events are retained in the workflow database artifact. PRs closed without merge are counted but not narrated. Open PRs do not carry into a new month without code or state activity in that month.