Found while producing MLA validation rows for #425 (after fixing #470 so autopatch actually engages).
Env: vLLM 0.24.0, kvcached main 60cad94 (editable install + kvcached_autopatch.pth copied into site-packages per #470), 1x L40S, driver 570.124.06, torch 2.11.0+cu130 with the cuda-compat-13 shim, ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1.
Repro: serve deepseek-ai/DeepSeek-V2-Lite-Chat (either KVCACHED_CONTIGUOUS_LAYOUT value). Patches apply (Applying 9 patches for vllm, Successfully patched vllm: ...), PageAllocator initializes (num_layers=27 ... num_kv_buffers=1), then post-init fails:
[kvcached][INFO] [kv_cache_manager.py:173] Set up broadcast callbacks for multi-process (world_size=1, use_worker_ipc=True)
[kvcached][ERROR][kv_cache_manager.py:234] Error during KVCacheManager post-initialization: Worker 0 failed to check KV tensors created: [Errno 2] No such file or directory
After that every allocation retries forever: alloc_page() failed after partially allocating 0/2 blocks; rolled back: Failed to map page 0: RuntimeError: Worker 0 failed to map: [Errno 2] No such file or directory (tp_ipc_util.py:217 _broadcast_map_to_kv_tensors) alternating with patches.py:2213 Shared physical KV pool is exhausted; reporting a scheduling miss so the engine can preempt and retry. The loop wrote >100 MB of warnings in ~8 min; zero requests ever complete.
Control in the same session and env: Qwen/Qwen2.5-0.5B-Instruct serves normally (RC 0, generation fine), so this is not general env breakage. The 0.5B log does not show the use_worker_ipc=True broadcast setup, so my read is the worker-IPC broadcast path itself is what fails: whatever should start the worker-side listener socket does not (or the manager computes a different socket path), and MLA (num_kv_buffers=1) is the path that requires it. Same failure with VLLM_ENABLE_V1_MULTIPROCESSING=0, so it is not the engine-process split.
Full logs kept; happy to bisect or test candidate fixes.
Found while producing MLA validation rows for #425 (after fixing #470 so autopatch actually engages).
Env: vLLM 0.24.0, kvcached main 60cad94 (editable install +
kvcached_autopatch.pthcopied into site-packages per #470), 1x L40S, driver 570.124.06, torch 2.11.0+cu130 with the cuda-compat-13 shim,ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1.Repro: serve deepseek-ai/DeepSeek-V2-Lite-Chat (either
KVCACHED_CONTIGUOUS_LAYOUTvalue). Patches apply (Applying 9 patches for vllm,Successfully patched vllm: ...), PageAllocator initializes (num_layers=27 ... num_kv_buffers=1), then post-init fails:After that every allocation retries forever:
alloc_page() failed after partially allocating 0/2 blocks; rolled back: Failed to map page 0: RuntimeError: Worker 0 failed to map: [Errno 2] No such file or directory(tp_ipc_util.py:217 _broadcast_map_to_kv_tensors) alternating withpatches.py:2213 Shared physical KV pool is exhausted; reporting a scheduling miss so the engine can preempt and retry. The loop wrote >100 MB of warnings in ~8 min; zero requests ever complete.Control in the same session and env: Qwen/Qwen2.5-0.5B-Instruct serves normally (RC 0, generation fine), so this is not general env breakage. The 0.5B log does not show the
use_worker_ipc=Truebroadcast setup, so my read is the worker-IPC broadcast path itself is what fails: whatever should start the worker-side listener socket does not (or the manager computes a different socket path), and MLA (num_kv_buffers=1) is the path that requires it. Same failure withVLLM_ENABLE_V1_MULTIPROCESSING=0, so it is not the engine-process split.Full logs kept; happy to bisect or test candidate fixes.