Initial setup eats 3Gb of memory on GPU0 (RTX3090, tp=8)

[Nepherpitou]

At weights loading moment vLLM already reserving ~3Gb of VRAM only on GPU0. This cause loss of 21Gb of total memory because tp=8 wants to have uniform memory across all 8 cards.

Sorry for lack of information, server is busy with building of vLLM at specific commit to verify issue.

Clean vLLM memory usage before model loading is ~400mb on each GPU. After loading - ~22.8Gb on each GPU.

Startup command and ENVs (llama swap)

  qwen3.5-122b-a10b-8bit:
    env:
      - VLLM_LOG_STATS_INTERVAL=5
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      - CUDA_DEVICE_ORDER=PCI_BUS_ID
      - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
      - OMP_NUM_THREADS=2
      - VIRTUAL_ENV=/home/gleb/llm/env_qwen_35
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
      - VLLM_NO_USAGE_STATS=1
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
      - VLLM_FLOAT32_MATMUL_PRECISION=high
      - VLLM_WORKER_MULTIPROC_METHOD=spawn
      - VLLM_MOE_USE_DEEP_GEMM=0
      - VLLM_USE_FLASHINFER_MOE_FP8=0
      - NCCL_P2P_DISABLE=1
      - CUDA_DEVICE_MAX_CONNECTIONS=8
      - GENESIS_ENABLE_P37=0
      - GENESIS_ENABLE_P61_QWEN3_MULTI_TOOL=1
      - GENESIS_ENABLE_P82=1
      - GENESIS_ENABLE_P61B_STREAMING_OVERLAP=1
      - GENESIS_ENABLE_PN13_CUDA_GRAPH_LAMBDA_ARITY=1
      - GENESIS_ENABLE_P59_QWEN3_TOOL_RECOVERY=1
      - GENESIS_ENABLE_PN8_MTP_DRAFT_ONLINE_QUANT=1
      - GENESIS_ENABLE_P81_FP8_BLOCK_SCALED_M_LE_8=1
      - GENESIS_ENABLE_P74_CHUNK_CLAMP=1
      - GENESIS_ENABLE_PN11_GDN_AB_CONTIGUOUS=1
      - GENESIS_BUFFER_MODE=shared
      - GENESIS_ENABLE_P95=1
      - GENESIS_ENABLE_P67_TQ_MULTI_QUERY_KERNEL=1
      - GENESIS_ENABLE_P58_ASYNC_PLACEHOLDER_FIX=1
      - GENESIS_ENABLE_PN50_GDN_FUSED_PROJ=1
      - GENESIS_ENABLE_P100=1
      - GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1
      - GENESIS_ENABLE_P70_AUTO_STRICT_NGRAM=1
      - GENESIS_P82_THRESHOLD_SINGLE=0.3
      - GENESIS_ENABLE_PN70_TOOL_SCHEMA_FILTER=1
      - GENESIS_ENABLE_P72_PROFILE_RUN_CAP=1
      #- VLLM_ENFORCE_STRICT_TOOL_CALLING=1
    cmd: |
      /home/gleb/.local/bin/uv run
        -m vllm.entrypoints.openai.api_server
        --model /mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-FP8
        #--model /mnt/data/llm-data/models/cyankiwi/Qwen3.5-122B-A10B-AWQ-8bit
        #--quantization humming
        --served-model-name "qwen3.5-122b-a10b-8bit"
        --port ${PORT}
        --tensor-parallel-size 8
        --enable-sleep-mode
        --max-num-batched-tokens 4096
        --enable-prefix-caching
        --enable-chunked-prefill
        #--kv-offloading-size 32
        --max-model-len auto
        --gpu-memory-utilization 0.9
        --max-num-seqs 4
        --attention-backend flashinfer
        --dtype half
        --reasoning-parser qwen3
        --enable-auto-tool-choice
        --tool-call-parser qwen3_coder
        #--chat-template /mnt/samsung_990_evo/llm-data/models/Qwen/qwen3.5-enhanced.jinja
        --chat-template /home/gleb/llm/patches/qwen_fixed_chat_template/chat_template_35.jinja
        #--load-format instanttensor
        #--enable-expert-parallel
        --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
    ttl: 0
    checkEndpoint: /v1/models

Clean vLLM startup log (Heavily truncated)

(APIServer pid=92107) INFO 05-08 16:17:21 [utils.py:306] 
(APIServer pid=92107) INFO 05-08 16:17:21 [utils.py:306]        █     █     █▄   ▄█
(APIServer pid=92107) INFO 05-08 16:17:21 [utils.py:306]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.2rc1.dev138+g52458b60a
(APIServer pid=92107) INFO 05-08 16:17:21 [utils.py:306]   █▄█▀ █     █     █     █  model   /mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-FP8
(APIServer pid=92107) INFO 05-08 16:17:21 [utils.py:306]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=92107) INFO 05-08 16:17:21 [utils.py:306] 
(APIServer pid=92107) INFO 05-08 16:17:21 [utils.py:240] non-default args: {'chat_template': '/home/gleb/llm/patches/qwen_fixed_chat_template/chat_template_35.jinja', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'port': 5008, 'model': '/mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-FP8', 'dtype': 'half', 'max_model_len': -1, 'served_model_name': ['qwen3.5-122b-a10b-8bit'], 'enable_sleep_mode': True, 'attention_backend': 'flashinfer', 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': True, 'max_num_batched_tokens': 4096, 'max_num_seqs': 4, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(APIServer pid=92107) INFO 05-08 16:17:21 [model.py:568] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=92107) WARNING 05-08 16:17:21 [model.py:2035] Casting torch.bfloat16 to torch.float16.
(APIServer pid=92107) INFO 05-08 16:17:21 [model.py:1697] Using max model len 262144
(APIServer pid=92107) INFO 05-08 16:17:22 [model.py:568] Resolved architecture: Qwen3_5MoeMTP
(APIServer pid=92107) WARNING 05-08 16:17:22 [model.py:2035] Casting torch.bfloat16 to torch.float16.
(APIServer pid=92107) INFO 05-08 16:17:22 [model.py:1697] Using max model len 262144
(APIServer pid=92107) WARNING 05-08 16:17:22 [speculative.py:672] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate
(APIServer pid=92107) INFO 05-08 16:17:22 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=92107) WARNING 05-08 16:17:22 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=92107) INFO 05-08 16:17:22 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=92107) INFO 05-08 16:17:22 [vllm.py:844] Asynchronous scheduling is enabled.
(APIServer pid=92107) INFO 05-08 16:17:22 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=92107) WARNING 05-08 16:17:22 [vllm.py:1412] max_num_scheduled_tokens is set to 4096 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
(APIServer pid=92107) INFO 05-08 16:17:23 [compilation.py:303] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=92107) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=92107) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=92344) INFO 05-08 16:17:40 [core.py:109] Initializing a V1 LLM engine (v0.20.2rc1.dev138+g52458b60a) with config: model='/mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-FP8', speculative_config=SpeculativeConfig(method='mtp', model='/mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-FP8', num_spec_tokens=3), tokenizer='/mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.5-122b-a10b-8bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto')
(EngineCore pid=92344) INFO 05-08 16:17:40 [multiproc_executor.py:139] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.1.6 (local), world_size=8, local_world_size=8
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(Worker pid=92379) INFO 05-08 16:17:50 [parallel_state.py:1410] world_size=8 rank=4 local_rank=4 distributed_init_method=tcp://127.0.0.1:59331 backend=nccl
(Worker pid=92375) INFO 05-08 16:17:52 [pynccl.py:111] vLLM is using nccl==2.28.9
(Worker pid=92375) WARNING 05-08 16:17:53 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available.
(Worker pid=92376) WARNING 05-08 16:17:53 [custom_all_reduce.py:153] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(Worker pid=92376) WARNING 05-08 16:17:54 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=92375) INFO 05-08 16:17:54 [topk_topp_sampler.py:45] Using FlashInfer for top-p & top-k sampling.
(Worker pid=92379) WARNING 05-08 16:17:54 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
(Worker pid=92376) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker_TP0 pid=92375) INFO 05-08 16:18:00 [gpu_model_runner.py:4858] Starting to load model /mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-FP8...
(Worker_TP1 pid=92376) INFO 05-08 16:18:00 [cuda.py:308] Using AttentionBackendEnum.FLASHINFER backend.
(Worker_TP7 pid=92382) INFO 05-08 16:18:00 [cuda.py:308] Using AttentionBackendEnum.FLASHINFER backend.
(Worker_TP0 pid=92375) INFO 05-08 16:18:00 [cuda.py:423] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP0 pid=92375) INFO 05-08 16:18:00 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP0 pid=92375) INFO 05-08 16:18:00 [__init__.py:393] Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod
(Worker_TP0 pid=92375) INFO 05-08 16:18:00 [gdn_linear_attn.py:168] Using Triton/FLA GDN prefill kernel
(Worker_TP0 pid=92375) INFO 05-08 16:18:00 [fp8.py:407] Using MARLIN Fp8 MoE backend out of potential backends: ['AITER', 'TRITON', 'MARLIN', 'BATCHED_TRITON', 'XPU', 'CPU'].
(Worker_TP0 pid=92375) INFO 05-08 16:18:01 [cuda.py:308] Using AttentionBackendEnum.FLASHINFER backend.
(Worker_TP0 pid=92375) 
(Worker_TP0 pid=92375) INFO 05-08 16:18:17 [default_loader.py:391] Loading weights took 16.19 seconds
(Worker_TP0 pid=92375) WARNING 05-08 16:18:17 [marlin_utils_fp8.py:97] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(Worker_TP0 pid=92375) INFO 05-08 16:18:18 [fp8.py:605] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP1 pid=92376) INFO 05-08 16:18:20 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0 pid=92375) INFO 05-08 16:18:20 [gpu_model_runner.py:4882] Loading drafter model...
(Worker_TP0 pid=92375) INFO 05-08 16:18:20 [vllm.py:844] Asynchronous scheduling is enabled.
(Worker_TP0 pid=92375) INFO 05-08 16:18:20 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0 pid=92375) WARNING 05-08 16:18:20 [vllm.py:1412] max_num_scheduled_tokens is set to 4096 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
(Worker_TP0 pid=92375) INFO 05-08 16:18:20 [compilation.py:303] Enabled custom fusions: norm_quant, act_quant
(Worker_TP0 pid=92375) INFO 05-08 16:18:20 [cuda.py:368] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=92375) INFO 05-08 16:18:20 [flash_attn.py:641] Using FlashAttention version 2
(Worker_TP0 pid=92375) INFO 05-08 16:18:20 [weight_utils.py:904] Filesystem type for checkpoints: XFS. Checkpoint size: 118.43 GiB. Available RAM: 161.22 GiB.
(Worker_TP0 pid=92375) INFO 05-08 16:18:23 [default_loader.py:391] Loading weights took 2.35 seconds
(Worker_TP5 pid=92380) INFO 05-08 16:18:23 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP2 pid=92377) INFO 05-08 16:18:23 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP1 pid=92376) INFO 05-08 16:18:23 [llm_base_proposer.py:1487] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP1 pid=92376) INFO 05-08 16:18:23 [llm_base_proposer.py:1543] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP3 pid=92378) INFO 05-08 16:18:23 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0 pid=92375) INFO 05-08 16:18:23 [llm_base_proposer.py:1487] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP0 pid=92375) INFO 05-08 16:18:23 [llm_base_proposer.py:1543] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP1 pid=92376) INFO 05-08 16:18:24 [interface.py:645] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP1 pid=92376) INFO 05-08 16:18:24 [interface.py:669] Padding mamba page size by 2.64% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP6 pid=92381) INFO 05-08 16:18:24 [llm_base_proposer.py:1487] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP6 pid=92381) INFO 05-08 16:18:24 [llm_base_proposer.py:1543] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP0 pid=92375) INFO 05-08 16:18:24 [gpu_model_runner.py:4960] Model loading took 15.26 GiB memory and 23.107530 seconds
(Worker_TP0 pid=92375) INFO 05-08 16:18:24 [interface.py:645] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP0 pid=92375) INFO 05-08 16:18:24 [interface.py:669] Padding mamba page size by 2.64% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP7 pid=92382) INFO 05-08 16:18:25 [llm_base_proposer.py:1487] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP7 pid=92382) INFO 05-08 16:18:25 [llm_base_proposer.py:1543] Detected MTP model. Sharing target model lm_head weights with the draft model.
(Worker_TP6 pid=92381) INFO 05-08 16:18:25 [interface.py:645] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP6 pid=92381) INFO 05-08 16:18:25 [interface.py:669] Padding mamba page size by 2.64% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP7 pid=92382) INFO 05-08 16:18:26 [interface.py:645] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP7 pid=92382) INFO 05-08 16:18:26 [interface.py:669] Padding mamba page size by 2.64% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP4 pid=92379) INFO 05-08 16:18:26 [llm_base_proposer.py:1487] Detected MTP model. Sharing target model embedding weights with the draft model.
(Worker_TP3 pid=92378) INFO 05-08 16:18:28 [interface.py:645] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP3 pid=92378) INFO 05-08 16:18:28 [interface.py:669] Padding mamba page size by 2.64% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=92375) INFO 05-08 16:18:28 [gpu_model_runner.py:5921] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(Worker_TP1 pid=92376) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP6 pid=92381) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP5 pid=92380) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP0 pid=92375) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP2 pid=92377) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP4 pid=92379) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP7 pid=92382) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP3 pid=92378) INFO 05-08 16:20:17 [gpu_model_runner.py:6143] Estimated CUDA graph memory: 0.25 GiB total
(Worker_TP1 pid=92376) INFO 05-08 16:20:17 [gpu_worker.py:475] CUDA graph memory profiling is enabled (default since 
(EngineCore pid=92344) WARNING 05-08 16:20:18 [kv_cache_utils.py:1152] Add 3 padding layers, may waste at most 8.33% KV cache memory
(EngineCore pid=92344) INFO 05-08 16:20:18 [kv_cache_utils.py:1863] Auto-fit max_model_len: full model context length 262144 fits in available GPU memory
(EngineCore pid=92344) INFO 05-08 16:20:18 [kv_cache_utils.py:1710] GPU KV cache size: 460,993 tokens
(EngineCore pid=92344) INFO 05-08 16:20:18 [kv_cache_utils.py:1711] Maximum concurrency for 262,144 tokens per request: 1.76x
(Worker_TP5 pid=92380) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP3 pid=92378) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP0 pid=92375) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP1 pid=92376) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP4 pid=92379) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP6 pid=92381) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP2 pid=92377) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP7 pid=92382) INFO 05-08 16:20:18 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled.
(Worker_TP0 pid=92375) 
(Worker_TP1 pid=92376) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(Worker_TP3 pid=92378) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(Worker_TP5 pid=92380) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(Worker_TP0 pid=92375) INFO 05-08 16:20:21 [gpu_model_runner.py:6242] Graph capturing finished in 3 secs, took 0.19 GiB
(Worker_TP0 pid=92375) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(Worker_TP6 pid=92381) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(Worker_TP2 pid=92377) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(Worker_TP4 pid=92379) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(Worker_TP7 pid=92382) INFO 05-08 16:20:21 [gpu_worker.py:619] CUDA graph pool memory: 0.19 GiB (actual), 0.25 GiB (estimated), difference: 0.06 GiB (33.7%).
(EngineCore pid=92344) INFO 05-08 16:20:22 [core.py:299] init engine (profile, create kv cache, warmup model) took 113.78 s (compilation: 10.00 s)
(EngineCore pid=92344) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=92344) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=92344) INFO 05-08 16:20:29 [vllm.py:844] Asynchronous scheduling is enabled.
(EngineCore pid=92344) INFO 05-08 16:20:29 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=92344) WARNING 05-08 16:20:29 [vllm.py:1412] max_num_scheduled_tokens is set to 4096 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
(EngineCore pid=92344) INFO 05-08 16:20:29 [compilation.py:303] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=92107) INFO 05-08 16:20:29 [api_server.py:613] Supported tasks: ['generate']
(APIServer pid=92107) INFO 05-08 16:20:30 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=92107) WARNING 05-08 16:20:30 [model.py:1454] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=92107) INFO 05-08 16:20:30 [hf.py:483] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=92107) INFO 05-08 16:20:48 [base.py:224] Multi-modal warmup completed in 18.000s
(APIServer pid=92107) INFO:     Started server process [92107]
(APIServer pid=92107) INFO:     Waiting for application startup.
(APIServer pid=92107) INFO:     Application startup complete.
(APIServer pid=92107) INFO:     127.0.0.1:44214 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=92107) INFO:     192.168.1.5:0 - "GET / HTTP/1.1" 404 Not Found
(APIServer pid=92107) INFO:     10.88.0.36:0 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=92107) INFO 05-08 16:21:30 [loggers.py:271] Engine 000: Avg prompt throughput: 581.9 tokens/s, Avg generation throughput: 17.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=92107) INFO 05-08 16:21:30 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.40, Accepted throughput: 0.80 tokens/s, Drafted throughput: 1.72 tokens/s, Accepted: 49 tokens, Drafted: 105 tokens, Per-position acceptance rate: 0.714, 0.429, 0.257, Avg Draft acceptance rate: 46.7%
(APIServer pid=92107) INFO 05-08 16:21:35 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 74.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=92107) INFO 05-08 16:21:35 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.41, Accepted throughput: 43.59 tokens/s, Drafted throughput: 92.98 tokens/s, Accepted: 218 tokens, Drafted: 465 tokens, Per-position acceptance rate: 0.690, 0.445, 0.271, Avg Draft acceptance rate: 46.9%
(APIServer pid=92107) INFO 05-08 16:21:40 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=92107) INFO 05-08 16:21:40 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.30, Accepted throughput: 41.19 tokens/s, Drafted throughput: 94.78 tokens/s, Accepted: 206 tokens, Drafted: 474 tokens, Per-position acceptance rate: 0.633, 0.411, 0.259, Avg Draft acceptance rate: 43.5%
(APIServer pid=92107) INFO 05-08 16:21:45 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 80.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=92107) INFO 05-08 16:21:45 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.51, Accepted throughput: 48.59 tokens/s, Drafted throughput: 96.58 tokens/s, Accepted: 243 tokens, Drafted: 483 tokens, Per-position acceptance rate: 0.752, 0.478, 0.280, Avg Draft acceptance rate: 50.3%
(APIServer pid=92107) INFO 05-08 16:21:50 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 77.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=92107) INFO 05-08 16:21:50 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.40, Accepted throughput: 45.39 tokens/s, Drafted throughput: 97.18 tokens/s, Accepted: 227 tokens, Drafted: 486 tokens, Per-position acceptance rate: 0.691, 0.463, 0.247, Avg Draft acceptance rate: 46.7%
(APIServer pid=92107) INFO 05-08 16:21:55 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 28.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=92107) INFO 05-08 16:21:55 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.20, Accepted throughput: 15.60 tokens/s, Drafted throughput: 38.99 tokens/s, Accepted: 78 tokens, Drafted: 195 tokens, Per-position acceptance rate: 0.615, 0.415, 0.169, Avg Draft acceptance rate: 40.0%
(APIServer pid=92107) INFO 05-08 16:22:00 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%

[Nepherpitou]

vLLM collect env

vllm collect-env
Collecting environment information...
uv is set
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : version 3.28.3
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.12 (main, Oct 31 2025, 23:02:31) [Clang 21.1.4 ] (64-bit runtime)
Python platform              : Linux-6.17.0-23-generic-x86_64-with-glibc2.39
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.1.115
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090
GPU 4: NVIDIA GeForce RTX 3090
GPU 5: NVIDIA GeForce RTX 3090
GPU 6: NVIDIA GeForce RTX 3090
GPU 7: NVIDIA GeForce RTX 3090

Nvidia driver version        : 595.58.03
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Архитектура:                             x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           43 bits physical, 48 bits virtual
Порядок байт:                            Little Endian
CPU(s):                                  128
On-line CPU(s) list:                     0-127
ID прроизводителя:                       AuthenticAMD
Имя модели:                              AMD EPYC 7702 64-Core Processor
Семейство ЦПУ:                           23
Модель:                                  49
Потоков на ядро:                         2
Ядер на сокет:                           64
Сокетов:                                 1
Степпинг:                                0
Frequency boost:                         enabled
CPU(s) scaling MHz:                      49%
CPU max MHz:                             3353,5149
CPU min MHz:                             1500,0000
BogoMIPS:                                3992,66
Флаги:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
Виртуализация:                           AMD-V
L1d cache:                               2 MiB (64 instances)
L1i cache:                               2 MiB (64 instances)
L2 cache:                                32 MiB (64 instances)
L3 cache:                                256 MiB (16 instances)
NUMA node(s):                            4
NUMA node0 CPU(s):                       0-15,64-79
NUMA node1 CPU(s):                       16-31,80-95
NUMA node2 CPU(s):                       32-47,96-111
NUMA node3 CPU(s):                       48-63,112-127
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.0
[pip3] nvidia-cutlass-dsl-libs-base==4.5.0
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0
[pip3] torch-c-dlpack-ext==0.1.5
[pip3] torchaudio==2.11.0
[pip3] torchvision==0.26.0
[pip3] transformers==5.8.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.20.2rc1.dev9+g01d4d1ad3 (git sha: 01d4d1ad3)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     48-63,112-127   3               N/A
GPU1    SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     32-47,96-111    2               N/A
GPU2    SYS     SYS      X      PHB     PHB     PHB     SYS     SYS     16-31,80-95     1               N/A
GPU3    SYS     SYS     PHB      X      PHB     PHB     SYS     SYS     16-31,80-95     1               N/A
GPU4    SYS     SYS     PHB     PHB      X      PHB     SYS     SYS     16-31,80-95     1               N/A
GPU5    SYS     SYS     PHB     PHB     PHB      X      SYS     SYS     16-31,80-95     1               N/A
GPU6    SYS     SYS     SYS     SYS     SYS     SYS      X      PHB     0-15,64-79      0               N/A
GPU7    SYS     SYS     SYS     SYS     SYS     SYS     PHB      X      0-15,64-79      0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_gleb
LD_LIBRARY_PATH=/home/gleb/llm/env_genesis/lib/python3.12/site-packages/cv2/../../lib64:
VLLM_WORKER_MULTIPROC_METHOD=spawn

[Sandermage]

Thanks for the detailed report and the full env/log dump — that helps a lot.

Short answer: this is upstream vLLM + CUDA/NCCL rank-0 overhead, not a Genesis
patch. I traced every Genesis patch you have enabled
(P61, P82, P61B, PN13, P59, PN8, P81, P74, PN11, P95, P67, P58, PN50, P100, P68,
P70, PN70, P72), and none of them allocates on GPU0 / rank-0 specifically:

• All Genesis buffer allocations are device-relative (device=self,
  torch.empty_like, etc.) — they follow the current rank's device, never a
  hardcoded cuda:0.
• Genesis patches are applied identically in every TP worker process (there is no
  rank == 0 gating in the apply path), so whatever a patch allocates, it
  allocates symmetrically across all 8 ranks. A symmetric patch can't be the
  source of an asymmetric GPU0 bump.
• P72_PROFILE_RUN_CAP (the only memory-relevant flag you have on) actually
  lowers profiling peak, uniformly, by <1 MB. Not a factor.

The ~3 GiB on GPU0 only is the classic device-0 / rank-0 collection that vanilla
vLLM also shows on this topology:

• CUDA primary context + cuBLAS/cuDNN/FlashInfer workspace handles,
• NCCL communicator buffers (rank-0 typically holds the rendezvous/root
  allocations; with NCCL_P2P_DISABLE=1 over a PCIe SYS/PHB topology like yours,
  NCCL stages through extra host-pinned/device buffers),
• PyTorch first-touch allocations that land on the default device.

Worth noting from your own log: weights are only 15.26 GiB/rank
(Model loading took 15.26 GiB), and the "~22.8 GiB after load" is
weights + CUDA context + KV. The 3 GiB you see is on top of that, on rank 0.

vLLM computes gpu_memory_utilization against per-rank free memory taken
after NCCL init, so the NCCL cost on the heaviest rank silently eats into the
uniform KV budget — which is exactly the "lose 21 GiB aggregate" symptom you
describe. This is an upstream accounting issue (RFC vllm#34303 / PR vllm#45517),
not something Genesis introduces.

Things to try

1. NCCL buffer sizing. You have NCCL_P2P_DISABLE=1. On a PCIe-only SYS/PHB
   topology that can increase host-staged buffers. Try:
   • NCCL_BUFFSIZE=2097152 (halve from the 4 MB default) and/or
     NCCL_MIN_NCHANNELS=1 NCCL_MAX_NCHANNELS=2 to shrink comms buffers, and
   • A/B test with P2P enabled if your board exposes it.
2. CUDA_DEVICE_MAX_CONNECTIONS=8 is non-default (default 1). Higher values
   allocate more per-stream scheduling resources. Revert to default and re-measure.
3. Move the GMU baseline before NCCL. Genesis ships a backport of vllm#45517:
   set GENESIS_ENABLE_PN517_INIT_SNAPSHOT_BEFORE_NCCL=1 and
   VLLM_INIT_SNAPSHOT_BEFORE_NCCL=1. It snapshots free memory before NCCL
   allocates and logs a per-rank _startup_free_bytes reading — which both (a)
   folds the NCCL cost into the GMU budget so KV stays uniform, and (b) tells you
   exactly how much each rank lost and when.

What would let us pin it down

We run 2× A5000 (TP=2), so we can't reproduce your 8× 3090 TP=8 122B config
directly — but we can confirm the mechanism from a per-rank breakdown. Could you
post:

• nvidia-smi --query-gpu=index,memory.used --format=csv at three points:
  before model load, right after weights load, and after KV alloc;
• the Genesis boot summary block from the log (the
  Genesis vLLM Patcher — boot summary section);
• ideally a run with PN517 enabled showing the per-rank _startup_free_bytes.

That isolates whether the 3 GiB lands before NCCL (CUDA context — pure upstream,
nothing to tune Genesis-side) or after (NCCL buffers — tunable via the env knobs
above). Happy to dig further once we see where in the boot timeline the GPU0 delta
appears.