A safetensors extension to efficiently store sparse quantized tensors on disk

TPU inference for vLLM, with unified JAX and PyTorch support.

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

This repo hosts code for vLLM CI & Performance Benchmark infrastructure.

Community maintained hardware plugin for vLLM on Spyre

A high-throughput and memory-efficient inference and serving engine for LLMs

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

Community maintained hardware plugin for vLLM on Ascend

Community maintained hardware plugin for vLLM on Intel Gaudi