[RFC] Initial Support for CPUs

[bigPYJ1151]

Progress

• Integrate CPU executor to support the basic model inference (BF16/FP32) without TP.
  • [Hardware][Intel] Add CPU inference backend #3634
  • [Hardware][Intel] Isolate CPUModelRunner and ModelRunner for better maintenance #3824
  • [CI/BUILD] enable intel queue for longer CPU tests #4113
  • [Hardware][Intel] Optimize CPU backend and add more performance tips #4971
  • [Hardware][Intel] Support CPU inference with AVX2 ISA #5452
  • [Hardware][Intel] Generate custom activation ops using torch.compile for CPU backend. #5446
• Support FP16 model inference.
• Support TP inference for multiple CPU sockets inside the same node.
  • [Hardware][Intel CPU] Adding intel openmp tunings in Docker file #6008
  • [Hardware] [Intel] Enable Multiprocessing and tensor parallel in CPU backend and update documentation  #6125
• Support model and KV cache quantization.
  • [Hardware][Intel] fp8 kv cache support for CPU #5492
  • [Hardware][Intel] Support compressed-tensor W8A8 for CPU backend #7257

Features

The CPU executor plans to support the following features:

• Basic models of vLLM with FP16/BF16/FP32, except MoE models
• Tensor-parallel model inference based on Ray
• AWQ quantization, 8-bit KVCache Quantization
• Others

Design

Our target is seamless porting vLLM to CPU devices and sharing most of vLLM core components (e.g., schedular, cache management, model definitions, Megatron-style model partitioning, ...).

The CPU executor will depend on Pytorch CPU and leverage optimized kernels and features from intel-extension-for-pytorch.

The main changes to vLLM include:

Torch APIs Adaption

CPU device is supported in PyTorch by default. It allows the CPU Executor to share the same model definitions with the GPU Executor. Thanks to recent code refactors, many hardcoded cuda device flags have been removed and Torch APIs are dispatched based on the device flag from DeviceConfig. For the CPU executor, a new cpu device flag is added.

Sharing the same model definitions and Torch APIs also allows the CPU executor to easily support new models and features in vLLM (e.g., torch.compile).

Custom Ops Adaption

vLLM implemented many efficient CUDA kernels and packaged as _C library by pybind. These kernels are ported to CPU using C++ and OpenMP, with the same function signatures to replace the CUDA kernels directly. The CPU custom kernel building procedure is integrated into vLLM CMake build system as a CMake module.

Currently, all of CPU kernels require AVX512 ISA support.

Python APIs Adaption

New CPUExecutor and CPUWorker are added to initialize the environment and model runner. The CPUModelRunner is derived from ModelRunner of the GPU code path, because most of the code could be shared. Even though it might have potential risks due to changes in the GPU code path, CPUModelRunner could fix them by rewriting configurations or overloading member functions easily.

In special, different from the GPU executor profiling available KV cache memory, the cache memory in the CPU executor is specified by the swap_space parameter. Because the memory management of CPU is more complex than GPU (e.g., NUMA).

[hyperbolic-c]

Thanks for yout excellent work! Looking forward to supporting inference on ARM CPU. Further, support for ray distributed computing
.

[hiGiraffe]

could you give me a cpu inference example?
I try

//start
python3 -m vllm.entrypoints.openai.api_server \
--device cpu

//input
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
        "model": "facebook/opt-125m",
        "messages": [
            {"role": "system", "content": "You are an intelligent British female writer and translator who is good at writing science fiction using multiple languages. You won a Nobel price in literature five years ago."},
            {"role": "user", "content": "Please detailedly tell a story about an exciting aerospace expedition for a Chinese boy Lam and his German dog. They are sent to aerospace by mistake and strive to wait for rescue from motherland with no water and food supply for over a month. They are almost caught by aliens disguised as his mother. Moreover, please translate the above story to Chinese, German, French, Portuguese and Japanese respectively."}
        ], "temperature": 0
    }'

But I got error. Are there any engine arguments that need to be added here?

[mgiessing]

@bigPYJ1151 Are you planning to support AVX/AVX2 to enable a broader range of Intel/x86 CPUs?

[bigPYJ1151]

Hi @mgiessing, it is not in our plan right now, but we may add it after the basic features finished.

[kannon92]

Could you help with #4415?

I was trying to compile cool with intel compiler but I had some issues and I think I almost have it working.

[hmellor]

Hi @bigPYJ1151,

I'd like to ask why the initial CPU support defines device specific vector types in https://github.com/vllm-project/vllm/blob/main/csrc/cpu/cpu_types_x86.hpp?

PyTorch contains a vector type Vectorized that appears to serve the same purpose, while also being architecture agnostic. Could the custom ops for CPU switch to using this PyTorch type to make the CPU backend architecture agnostic? (i.e. PowerPC, AArch64, etc.)

[bigPYJ1151]

Hi @hmellor

Yes, Pytorch contains such vector structures, and it is feasible to use them in the CPU backend. I didn't aware them before so defined the custom types🤣. vLLM is adapting torch compile and some custom ops will be generated by JIT, so the number of custom types will be very limited after we clean them. Then we can try to replace them with the Pytorch vectors.

[hmellor]

That's great to hear! Is it just #7110 that we're waiting for, or are there other PRs?

[bigPYJ1151]

Yes, after the #7110 I think we can do some code refactors.