Shared-GPU startup memory budgeting with kvcached

[shipiyouniao]

Status

Settled for now. Reopen this design discussion only if production evidence shows that the current compatibility solution is insufficient.

The current direction is the stable logical KV capacity implemented in #448. A complete redesign of inference-engine memory profiling is not part of that fix.

Context

Inference engines such as vLLM derive KV capacity from whole-device GPU memory snapshots. The calculation assumes that other processes sharing the GPU do not change their memory usage during profiling.

That assumption does not hold for colocated engines using kvcached. Model loading, CUDA contexts, compilation, CUDA Graph capture, workspaces, and physical KV page mappings all consume or release physical GPU memory. A peer process can therefore change whole-device free memory while another engine profiles.

Observed effects include:

• peer growth being attributed to the profiling engine, which underestimates KV capacity or produces no usable cache blocks;
• peer release increasing whole-device free memory during profiling and triggering vLLM's consistency error;
• identical SGLang configurations receiving order-dependent logical KV capacities when they sample device free memory at different times.

In one unfixed vLLM run, free memory changed from 8.57 GiB to 12.91 GiB during profiling and terminated EngineCore. In a T4 SGLang reproduction, two identical instances received 1,048,596 and 832,174 logical tokens respectively, a 20.6% difference caused only by startup order.

Options considered

1. Serialize startup profiling

A startup lock can stop two initializers from profiling simultaneously, but it does not restore the whole-device snapshot assumption:

1. Engine A finishes initialization and releases the lock.
2. A begins serving and changes its physical memory usage.
3. Engine B profiles while A continues changing memory.

The same issue returns when a later engine C starts. Making serialization complete would require a global quiescence barrier over running engines, which would stall traffic and make cold-start latency grow with the number of colocated instances.

This option is not selected.

2. Use a stable logical KV capacity in the kvcached integration

This is the selected compatibility boundary and the implementation in #448:

• vLLM derives logical KV capacity from total device memory and gpu_memory_utilization;
• SGLang derives logical KV capacity from total device memory and mem_fraction_static;
• vLLM uses its existing explicit kv_cache_memory_bytes path, which still performs the required compilation profile;
• explicit user capacity remains authoritative;
• behavior is unchanged when kvcached is disabled;
• physical KV pages remain lazy and are allocated by kvcached on demand;
• temporary physical pressure during null-block reservation waits and retries rather than terminating the engine.

After the SGLang fix, both T4 instances received the same 14,073,377,588-byte logical capacity and 1,145,294 logical tokens, reached health, and completed requests.

This option covers both peer allocation and peer release without pretending that whole-device deltas are process-local accounting.

3. Redesign shared-GPU memory budgeting in the inference engine

A complete design could separate logical KV capacity from physical budgets for weights, activations, workspaces, communication buffers, CUDA contexts, CUDA Graphs, and other non-KV allocations.

This would be a larger inference-engine API and accounting change. It is easier to get wrong across versions and execution modes, so it is deliberately deferred. It should only proceed with a clear upstream contract and evidence that option 2 is insufficient.

Known boundary

The selected solution does not make a permanently infeasible configuration runnable. Physical non-KV headroom and at least one KV page must eventually be available.

When physical KV capacity never becomes available, null-block reservation continues waiting instead of failing fast. The retry is backed off and does not busy-spin, but the engine does not become usable. A future change may make this wait lifecycle-aware or connect it to an explicit caller-provided startup deadline without reintroducing timing-dependent failures.

Observability should expose allocator state for diagnosis and verification, but metrics and exported snapshots must not participate in allocation decisions.

Reopen criteria

Reopen this design discussion only when there is reproducible evidence of at least one of the following:

1. fix: decouple virtual KV capacity from shared-GPU profiling #448 still permits a startup failure caused by another process changing whole-device memory during profiling.
2. Stable logical capacity produces incorrect or inconsistent sizing on a supported inference-engine path.
3. A feasible configuration enters a persistent physical-capacity wait that cannot be addressed through allocator lifecycle or an explicit startup deadline.
4. A supported workload requires a process-local physical non-KV budget that the compatibility boundary cannot represent safely.

Performance differences or failures caused by a physically infeasible deployment are not, by themselves, evidence that the profiling redesign is required.

References

• start mutiple models #193: original same-GPU multi-instance startup report and design discussion
• fix: decouple virtual KV capacity from shared-GPU profiling #448: selected compatibility implementation and validation
• Add read-only observability snapshots #385 and fix: synchronize page allocator snapshots #443: stable pool identity and coherent allocator snapshots used for diagnosis
• Add KV cache operation observability #410 and feat: add revisioned instance memory limits #414: operation observability and physical-residency control work