RPC timeout error after 5 min hang with speculative config arctic-suffix=True

[jeong-tae]

I'm testing the speed of Llama-3.1-SwiftKV-8B-Instruct with aiperf. Currently, I'm testing with an input length of 32 and an output length of 30. As I gradually increase the concurrency, the test hangs at some point, and after 5 minutes, an RPC timeout error occurs.

When the hang occurs, message shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation). appears.

Setting the suffix option to false makes it a bit slower, but works well regardless of concurrency.

Any ideas?

[jeong-tae]

It happens under below settings

MODEL=Snowflake/Llama-3.1-SwiftKV-8B-Instruct
export CUDA_VISIBLE_DEVICES=0,1
export ARCTIC_INFERENCE_ENABLED=1

vllm serve $MODEL \                                                             
    --quantization "fp8" \                                                      
    --tensor-parallel-size 1 \                                                  
    --ulysses-sequence-parallel-size 2 \                                        
    --enable-shift-parallel \                                                   
    --gpu-memory-utilization 0.7 \                                              
    --shift-parallel-threshold 256 \                                            
    --max-model-len 65536 \                                                      
    --speculative-config '{                                                     
        "method": "arctic",                                                     
        "model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,                                            
        "enable_suffix_decoding": true,                                         
        "disable_by_batch_size": 64                                             
    }' \                                                                        
    --port 18099