Megatron-LM Monthly Activity Report — August 2026

[lhb8125]

Megatron-LM Monthly Activity Report — August 2026

Period: 2026-08-01 through 2026-08-31 (Asia/Shanghai); source: NVIDIA/Megatron-LM.

Overview

The delivered portfolio emphasizes Megatron-FSDP v2, GTP, dynamic inference, MoE execution, sparse attention, hybrid architectures, THD sequence packing, multimodal workloads, optimizer efficiency, and RL rollout reliability. Ongoing work extends these directions through finer-grained FSDP overlap, DeepSeek-V4 attention, disaggregated serving, new MoE kernels, hybrid sequence models, Muon variants, multimodal execution, and reproducibility infrastructure.

• PRs opened: 743; merged during the period: 397; closed without merge: 221; still open at period end: 322.
• Excluded from the narrative: test-only 41, CI-only 44, test/CI-only 0, format-only 0.

Delivered

Megatron-FSDP v2: Sharding, Optimizer, and Distributed Checkpoints

Megatron-FSDP v2 gained a dedicated optimizer, multiple data-parallel sharding strategies, independent expert sharding, distributed checkpoint support, and tighter gradient and storage lifecycle handling.

• No-shard and ZeRO-1/2/3 layouts now cover replicated and progressively sharded parameters, gradients, and optimizer states, with a separate policy available for expert parameters.
• Distributed checkpoint handling covers uneven DTensor shards, optimizer-state materialization, compute-weight synchronization, and fused MLA and MTP parameter mappings.
• Expert-gradient scaling, stream-owned main-gradient buffers, DTensor-aware clipping, and stable post-step weight synchronization address correctness across expert and dense meshes.
• Persistent communication-buffer pools and weak-reference ownership improve combined-1F1B capacity management and allow sharded-model CUDA storage to be released.

Related PRs: #5865, #5956, #6024, #6137, #6187, #6230, #6271, #6316, #6336, #6489, #6798

GTP: CUDA-Graph Communication, MTP Weight Reuse, and Checkpoint Resumption

Generalized tensor parallelism was hardened for CUDA-graph replay, repeated MTP parameter use, mixed-precision reduction, evaluation, Muon integration, and heterogeneous checkpoint resumption.

• Per-graph communication ownership restored reduce-scatter overlap while fixing weight-gradient buffer lifetime and stale-gradient races.
• Repeated MTP and recomputation consumers now retain distinct communication tickets or gather buffers, preventing missing gradients and weight corruption.
• Parameter-readiness callbacks and main-gradient dtype propagation prevent stale weight gathers and BF16 rounding before FP32 reduction.
• GTP-aware evaluation, heterogeneous MIMO checkpoint groups, and MXFP8-aware Muon gathering improve correctness across rematerialized and low-precision configurations.

Related PRs: #6060, #6242, #6388, #6407, #6624, #6717, #6743, #6754

Disaggregated Dynamic Inference: KV/SSM Handoff, Prefix Caching, and ReFIT

Dynamic inference added disaggregated KV and SSM state transfer, stronger prefix-cache lifecycle handling, asynchronous scheduling, progressive tool calls, and model-refit support.

• Asynchronous prefill-to-decode handoff transfers attention KV state and hybrid SSM state over NIXL or NCCL while protecting cache blocks and avoiding prompt replay on decode workers.
• Prefix-cache admission, KV-write isolation, GDP state reuse, and Mamba edge-case alignment improve behavior under saturation and chunked prefill.
• NCCL M-to-N refitting and in-place MXFP8 updates support topology changes while preserving CUDA-graph-captured storage; eager communicator initialization prevents idle-rank hangs.
• The serving stack also gained an inference metadata ledger, streamed tool-call deltas, default asynchronous scheduling, subscriber rendezvous, and server-controlled chat-template handling.

Related PRs: #5995, #6125, #6221, #6222, #6226, #6584, #6711, #6712, #6779, #6780, #6818, #6826, #6851, #6895, #6930, #6955

MoE Batch Invariance, CUDA Graphs, and MXFP8 Decode

MoE execution expanded across batch-invariant routing, graph-compatible dispatch, paged activation storage, MXFP8 communication and refitting, and device-initiated expert kernels.

• Batch-invariant execution now spans Nemotron routing, dispatch, fused experts, SSM kernels, and dynamic inference; the fused-MoE path reported roughly 25,100 tokens/s versus roughly 4,070 previously and roughly 25,400 without invariance.
• Paged stash supports whole-MoE CUDA graphs, while shared vLLM grouped-GEMM buffers reduced reported graph memory from more than 20 GB to approximately 3 GB.
• The NCCL dispatcher supports dynamic eager capacity and static graph capacity with overflow recovery, plus BF16 or MXFP8 token movement and an optional FP32 NVLS combine.
• Device-initiated GroupedLinear removes host token-count synchronization, and FlashInfer routed MXFP8 decode reported 35.2% lower time per token and 1.54× end-to-end throughput.

Related PRs: #4871, #6022, #6229, #6432, #6521, #6524, #6588, #6618, #6806, #6847, #6864, #6870

DeepSeek-V4 CSA/DSA: THD Kernels, Teacher Loss, and Fused RoPE

Compressed and sparse attention gained a native SBHD reference, corrected teacher-loss semantics, fused THD compression and RoPE paths, and more efficient sparse-indexer backward execution.

• Teacher distributions now use the complete CSA denominator and full attention LSE while excluding padded THD entries and compacting shared attention indices.
• The unfused BF16 SBHD implementation covers compression, indexing, sparse and dense attention, sliding windows, attention sinks, and teacher-loss normalization.
• Fused THD pooling reduces the documented eager launch sequence to one compute kernel per direction, while fused MLA/DSA RoPE reduced a targeted long-sequence region from 85 kernels to 2.
• Sparse-indexer padding changes reduced the reported kernel from about 12.824 ms to 1.273 ms, with median iteration latency down 9.9% and throughput up 11.0% on the stated proxy.

Related PRs: #5960, #5984, #6166, #6279, #6343, #6349, #6400, #6842, #6846

Hybrid Models and CUDA-Graph Recomputation: mHC, GDP/GDN, and KDA–MLA

Hybrid-model support grew through manifold hyper-connections, Gated Delta Product and Gated DeltaNet execution, Ling-V3 Tiny KDA–MLA training, and broader CUDA-graph and recomputation coverage.

• mHC gained core and fused implementations, selective recomputation, hybrid-model integration, and FP32 cuTile mapping; the reported residual-mapping error improved from 1.89e-3 to 1.09e-5.
• Gated Delta Product now spans training, static and dynamic inference, TP/CP, packed sequences, checkpointing, optimizer integration, activation controls, CuTe kernels, CUDA graphs, and chunkwise context parallelism.
• Ling-V3 Tiny support covers its 7.9B-parameter sparse-MoE KDA–MLA architecture, while Gated DeltaNet gained dynamic inference and recurrent-mixer execution.
• Graph and recomputation fixes cover aliased checkpoint tensors, expert-overlap layer segments, nested checkpoint parameters, deterministic stream assignment, mHC capture behavior, and partial hybrid MTP capture.

Related PRs: #4497, #4531, #5841, #5869, #6074, #6172, #6311, #6373, #6401, #6429, #6499, #6553, #6556, #6583, #6595, #6598, #6643, #6661, #6704, #6815

THD Sequence Packing and Hybrid Context-Parallel Layouts

Sequence-packed training gained reusable scheduling, THD batch distribution, MTP rolling, data-parallel balancing, and per-layer conversion between contiguous and zigzag context-parallel layouts.

• Core and data-parallel schedulers now pack, reroute, broadcast, and fetch variable microbatches with PackedSeqParams and physical padding metadata.
• Packed context-parallel MTP prefetches successor rows once per microbatch and reuses them across prediction depths.
• Data-parallel rerouting avoided fully connected NCCL P2P allocation and reduced reported memory by 1,152–1,154 MiB per GPU in a four-rank test, at the cost of greater communication volume.
• Qwen3.5 hybrid layouts reported a 1.29× overall proxy speedup, while MagiAttention adds attention-area-balanced context parallelism for variable-length packed Qwen3 MoE SFT.

Related PRs: #5903, #5904, #6246, #6378, #6387, #6478, #6515, #6625, #6626, #6684, #6764, #6776

Bagel and MIMO Multimodal Training with Image/Video Inference

Multimodal coverage expanded through Bagel training, dynamic image and video inference, MTP modality masking, heterogeneous MIMO parallelism, and per-modality precision control.

• Bagel support includes MCore and Hugging Face LLM wrappers, a ViT wrapper, diffusion components, data handling, checkpoint utilities, and training integration.
• Dynamic batching now accepts image-bearing requests, and inference APIs and OpenAI-compatible endpoints also accept video bytes, frame manifests, or preprocessed tensors.
• MTP masking excludes modality embeddings and cumulatively blocks prediction paths after image, audio, or other placeholders.
• MIMO gained dense and expert GTP topology, heterogeneous DDP overlap, batched pipeline communication, and independent precision policies for modality submodules.

Related PRs: #3635, #6010, #6256, #6260, #6284, #6570, #6606, #6745, #6770, #6809

Muon Shard Balancing and Chunked Optimizer Offload

Optimizer work reduced GPU residency and improved layer-wise parameter assignment through pinned CPU storage, bounded state chunks, shard-aware packing, and per-weight Muon execution choices.

• Chunked optimizer-state and master-weight offload replaces full-state GPU restoration with bounded chunks backed by pinned CPU storage.
• Re-pinning FP32-converted offloaded parameters preserves pinned-memory host-to-device transfers instead of falling back to pageable memory.
• Layer-wise bucket packing removed the documented 25% slot overhead, while compute-cost-aware Muon balancing improved a reported 1,024-GB200 workload from 2,080 to 1,707 ms per iteration.
• Explicit Muon ownership exclusions and automatic per-weight TP mode selection prevent unowned matrices and adapt Newton–Schulz execution to shape, precision, group size, and hardware costs.

Related PRs: #4547, #5415, #6244, #6379, #6642, #6664

RL Context Parallelism, Unified Rollouts, and Durable Recovery

Reinforcement-learning execution gained context-parallel packed sequences, shared prefill, a unified rollout pipeline, durable rollout storage, and recovery protections for infrastructure failures.

• Context parallelism now spans RL utilities, packed-sequence handling, Transformer Engine integration, and the training entry point, while shared prefill allows packed samples to reuse common prompt work.
• A unified pipeline centralizes rollout generation and scheduling across environments, with automatic lag tuning reducing manual configuration.
• Checksummed, fsync-backed rollout storage supports torn-write recovery, restoration, consumption tracking, and checkpoint-boundary compaction.
• Infrastructure-failure placeholders are excluded from GRPO rewards, and all-placeholder rollout groups are detected and regenerated.

Related PRs: #4127, #4362, #5882, #5887, #6352, #6475, #6480, #6495

In Progress

Megatron-FSDP v2: 1F1B Overlap, Expert Sharding, and CPU Offload

Ongoing Megatron-FSDP v2 work is developing fine-grained 1F1B lifecycle control, optimizer checkpoints, MXFP8 primary weights, expert-aware shard planning, CPU offload, and more robust asynchronous checkpoint handling.

• VPP and combined-1F1B paths are being wired with explicit unshard, reshard, reduction, prefetch, no-sync, delayed weight-gradient, shared-context, UBR, and cross-stream lifetime controls.
• Optimizer checkpoints, GroupedTensor expert storage, composed EP/DP placements, MXFP8 payloads, reproducible re-quantization, and reusable shard plans extend low-precision and expert workflows.
• Routed-expert weights and chunked Adam moments, master weights, initialization, and checkpoint saves are being moved through pinned CPU storage to bound GPU residency.
• A reported hybrid outer-optimizer strategy reached 1,450 TFLOP/s/GPU versus 1,195 for uniform ZeRO-3, while an initialization-stream change reduced reported reserved and device memory by 20.47 GiB.

Related PRs: #6197, #6215, #6228, #6484, #6485, #6486, #6494, #6544, #6594, #6597, #6665, #6666, #6694, #6773, #6785, #6885, #6948, #6971

GTP: On-Demand Materialization, Process Groups, and Persistent CUDA-Graph Buffers

Generalized tensor parallelism is being extended as an end-to-end weight-sharding system with explicit process groups, asynchronous materialization, fixed-address graph storage, repeated-parameter replay, and ReFIT support.

• The proposed GTP stack spans training, optimization, checkpoints, inference, CUDA graphs, low-precision parameters, MIMO, and MoE integration.
• Explicit process-group collections are being required across language models and related MCore paths to prevent collectives from silently using the wrong parallel grid.
• Persistent arenas replace specialized gradient rings, size storage by domain and dtype, and protect reusable CUDA-graph generations with replay-completion events.
• Repeated parameter use records every gradient-ready notification, while rematerialized tied embeddings and inference shards are being repaired for reshard and ReFIT workflows.

Related PRs: #6293, #6303, #6713, #6862, #6867, #6882, #6940

DeepSeek-V4 CSA/DSA: Hybrid Attention, MLA Latent CP, and NVFP4

DeepSeek-V4 work is integrating native and fused compressed sparse attention, GQA-based DSA, per-layer context layouts, MLA latent context parallelism, tensor and sequence parallelism, and NVFP4 training.

• Hybrid-attention orchestration and layer construction connect CSA, window attention, grouped output projection, MTP identity, indexer loss, and validated compression-ratio semantics.
• The fused SBHD CSA path targets SM90-or-newer GPUs, while GQA-based DSA adds reference and Triton implementations with memory-oriented execution.
• Packed-THD layout metadata, per-layer conversions, dynamic-CP graph sharing, communicator reuse, and DSA state reuse across MTP steps target lower repeated layout and indexer work.
• MLA latent CP, shared K-RoPE gradients, TP-plus-CP execution, NVFP4 training, compilation caches, and pipeline prewarming are being developed; the reported prewarm test reduced startup through the first iteration from 866.0 to 370.8 seconds.

Related PRs: #6206, #6233, #6397, #6402, #6404, #6405, #6426, #6472, #6479, #6699, #6801, #6829, #6839, #6841, #6848, #6850, #6869, #6874, #6887

MoE Decode Kernels, SeqTopK Routing, and Batch-Invariant Reduction

Ongoing MoE work targets sequence-level routing, fused decode kernels, graph-safe expert-parallel initialization, new execution backends, batch-invariant reduction, load balancing, and reusable activation buffers.

• SeqTopK proposes a fixed sequence-level expert budget with a configurable per-token cap, while dense routing indices avoid reconstructing supported Flex-dispatch inputs.
• Decode optimizations fuse FC1 activation, router selection, Q/K RMSNorm, routing-table construction, and padding handling and add separate FC1/FC2 tiling; individual reported gains range from kernel-level reductions to double-digit end-to-end improvement.
• MOK and Sonic-MoE add experimental fused backends, while device-initiated GroupedTensor experts are being integrated with paged stash and reusable HybridEP buffers.
• Batch-invariant NVLS reduction, global-batch quantile balancing, SiTU-GLU, grouped-MLP synchronization, allocator hand-back, and DDP-preserved fused expert views address training and colocated RL execution.

Related PRs: #6208, #6412, #6452, #6453, #6454, #6456, #6457, #6458, #6459, #6572, #6601, #6614, #6615, #6634, #6637, #6638, #6654, #6673, #6674, #6800, #6828, #6845, #6883, #6963

Disaggregated Inference, Prefix-Affinity Routing, and ReFIT Safety

Inference work is developing independently scaled prefill and decode workers, direct state handoff, prefix-affinity routing, lease-based cache management, asynchronous scheduling, and safer model refits.

• The disaggregated path targets direct NCCL or NIXL transfer of KV and SSM state between independently parallelized prefill and decode shards.
• Weight-version-salted prefix hashes, load-aware affinity routing, cache leases, and lifecycle fixes aim to prevent stale reuse and coordinate cache protection and eviction.
• Local-copy stream ordering, Mamba cache refresh, and direct BF16-to-MXFP8 conversion into persistent buffers address stale parameters and refit allocation overhead.
• Data-parallel HTTP frontends, ZMQ subscription rendezvous, asynchronous scheduling, request cancellation, and ReplaySSM speculative rollback extend serving scalability and resilience.

Related PRs: #6196, #6223, #6497, #6617, #6668, #6675, #6715, #6723, #6823, #6852, #6853, #6916, #6932, #6952

GatedDeltaNet, Mamba2, and Wide/Attention Residual Hybrid Models

Hybrid architecture work is adding native and fused GatedDeltaNet kernels, Mamba2 state-passing context parallelism, wide residual streams, Attention Residuals, KDA–MLA variants, and broader CUDA-graph compatibility.

• Internal CuTe and Transformer Engine paths provide experimental native and fused GatedDeltaNet execution, including replay-safe variable-length THD capture.
• Mamba2 context parallelism exchanges convolution halos and recurrent-state summaries, while selective replay reduces retained wide-boundary activations.
• Streamwise wide residuals keep attention, MLP, and MoE branches at ordinary width and are being composed with MTP, MIMO, Mamba, and mixed HybridStack models.
• Attention Residuals, KDA low-rank projections, MLA NOPE, runtime CP groups, allocator stream affinity, and full-iteration graph fixes expand hybrid model choices and graph coverage.

Related PRs: #6511, #6645, #6716, #6796, #6805, #6808, #6821, #6827, #6840, #6855, #6877, #6958

Muon Head-Wise, Hyperball, and Layer-Sharded Tensor Parallelism

Muon development is adding head-wise QKV handling, Hyperball projection, per-domain execution modes, gated-projection correctness, deterministic parameter-group ordering, and layer-sharded Newton–Schulz work.

• Head-wise orthogonalization covers MHA, GQA, gated attention, MLA, fragmented TP, and GTP-rematerialized layouts while excluding padding from head boundaries.
• TensorParallelMuonHT introduces fixed-radius initialization and Hyperball updates with logical Frobenius norms across TP, expert-TP, and GTP shards.
• Layer-sharded Muon assigns each matrix to one home rank; the reported optimizer step fell from 264 ms to 139 ms and NCCL calls fell to four at the stated production shapes.
• Independent dense and expert mode selection, gated-FC1 splitting, and stable expert-aware parameter ordering address execution choice and collective consistency.

Related PRs: #6326, #6422, #6616, #6683, #6688, #6761

MTP Alignment, Variable-Length THD Packing, and Durable RL Rollouts

Ongoing work is aligning MTP across packed and context-parallel layouts, completing variable-length THD training, reducing RL logit materialization, and preserving partial rollout progress.

• Absolute-offset roll contexts and CP-group boundary exchanges aim to keep MTP inputs, targets, losses, and CUDA-graph values aligned across local, contiguous, zigzag, and dynamic layouts.
• A draft-only total-variation MTP objective is being added alongside fused kernels, while variable-length sequence packing is being wired through training and evaluation.
• Vocabulary-parallel selected-token log probabilities avoid full-vocabulary logits for eligible RL consumers.
• Per-rollout ledgers preserve completed members of partial groups, while FSDP and Transformer Engine microbatch-state fixes prevent duplicate finalization and accumulated-gradient overwrites.

Related PRs: #6205, #6473, #6605, #6706, #6741, #6742, #6746, #6768, #6875, #6876, #6965

Qwen3.5-VL, Nemotron Omni, and MIMO Multimodal Execution

Multimodal work is extending Qwen3.5-VL training and GTP, Nemotron image/video/audio inference and SFT masking, dynamic-resolution visual processing, and configurable MIMO placement and communication.

• Qwen3.5-VL is being migrated to HybridModel with multimodal RoPE, Energon data loading, packed-sequence and CP handling, and generalized tensor parallelism.
• Dynamic-resolution pixel shuffling uses actual patch grids and video tubelets to align visual feature ordering between policy training and inference.
• The Nemotron Omni draft targets fixed-address image, video, and audio embeddings across paged caching, chunked prefill, and CUDA graphs, but the input states that it has not been executed or tested.
• MIMO work adds optional projector placement, bridge communication policies, modality-aware checkpoint replica IDs, and Nemotron SFT masking for assistant responses and retained thinking traces.

Related PRs: #6193, #6255, #6308, #6315, #6424, #6608, #6792, #6843, #6888

Kernel Backend Selection, Tensor Metrics, and Determinism Tracing

Execution infrastructure is being developed for per-operation backend selection, distributed tensor diagnostics, rank-local determinism tracing, and reproducible low-level execution behavior.

• A rank-local JSONL tracer and offline comparator provide metadata, summaries, samples, and full-tensor evidence without distributed collectives.
• Tensor observation hooks, distributed metric definitions, and training-loop integration expose parameters, gradients, residuals, logits, and router data through TensorBoard, Weights & Biases, or stdout.
• A centralized backend registry defines availability, determinism, fusion declarations, defaults, and per-operation overrides, replacing duplicated model-spec selection logic.
• Rank-local RNG checkpoint offsets, corrected batch-invariant RMSNorm, explicit Triton cache requirements, and independently controlled MoE auxiliary-loss fusion target reproducible resume and execution.

Related PRs: #6610, #6647, #6648, #6649, #6650, #6689, #6690, #6766, #6858, #6859, #6966, #6967

Scope and Method

The narrative is grouped by technical theme. Related dev and main PRs are described once; the complete PR ledger, exclusion evidence, and state events remain in the local database. PRs closed without merge are counted but not narrated.