Development Roadmap (2024 Q4)

[Ying1123]

Here is the development roadmap for 2024 Q4. Contributions and feedback are welcome (Join Bi-weekly Development Meeting). Previous 2024 Q3 roadmap can be found in #634.

Performance

• Hide CPU overhead with overlapped scheduler (Faster overlap mode scheduler #1738)
• Support speculative decoding
  • Eagle [WIP] Spec infer with EAGLE2 #1498
  • Reference-based. Reference speculative decoding #270
  • Medusa head [Feature] plan to support medusa? #859
  • Draft model based.
• Sparse Attention Support double sparsity #1459
• Faster grammar parsing library for constrained decoding [Performance] Support both xgrammar and outlines for constrained decoding #1752
• Multi-layer radix cache (GPU/CPU/Disk) @xiezhq-hermann

Parallelism

• Finish sequence parallelism [Feature] Add initial support for sequence parallelism #1436
• Support pipeline parallelism.
• Support expert parallelism + data parallelism for DeepSeek/MoE models. @ispobock
• Implement a better cache-aware load balancer for data parallelism. See also preble. [router] rust-based router #1790 @ByronHsu @yichuan520030910320
• Overlap communication in tensor parallelsim. @ZhuohaoL
• Support disaggregated serving to separate prefill and decoding.

Hardware Coverage

• AMD optimizations. cc @HaiShaw
  • FlashInfer support for AMD
  • Setup CI (accuracy/performance) for AMD
• Intel XPU support.
  • [Feature, Hardware] Enable SGLang on XPU GPUs via PyTorch #1480

Model Coverage

• Multi-modal models
  • Llama 3.2 Vision Llama3.2 vision model support #1551
  • QWen2-VL Support qwen2 vl model #1546
  • GLM 4V Add GLM-4v Multimodal Model support for SGLang #1641
  • VILA
  • Phi-vision
  • FishSpeech audio model support
  • Ultravox
• Language models
  • Mamba models
• Reward models
  • [Feature] Support reward model LxzGordon/URM-LLaMa-3.1-8B #1525

LoRA support

• Performance optimizations for multi-LoRA serving [LoRA, Performance] Add gemm expand triton kernel for multi-LoRA #1728

LMCache Integration

• Integrate with LMCache https://github.com/LMCache/LMCache

Quantization

@HaiShaw @zhyncs @ispobock

• Torchao integration Add llama implementation with no tensor parallel linears #1561
• Turbomind operators integration
• More CUTLASS mixed precision gemm integration
• KV cache quantization (more formats + scaling factor)

Server API

• Support directly taking embedding as inputs. [Feature] Generation Inputs: input_embeds #745
• Add APIs for using the inference engine in a single script without launching a separate server. See also examples.
  • Provide an offline engine API #1567
• Support endpoint other than OpenAI (Anthropic, Mistral) in the language frontend.
• Better APIs to support RL trainers, including https://github.com/huggingface/trl and https://github.com/OpenRLHF/OpenRLHF @zhaochenyang20
• Support generalized reward API (adding linear layers to any Causal LM to get the reward) https://github.com/OpenRLHF/OpenRLHF @zhaochenyang20

Observability

• Integrate Grafana / Prometheus
  • support prometheus metrics #1853 [WIP] Prometheus Metrics #1461

Others

• Notebook-style interactive tutorials. @zhaochenyang20
• Compiler mode optimizations for the language (e.g. support sending a full serialized SGL program to the server). @hnyls2002
• Memory pool refactor to better support mixing different attention layers (e.g., interleaved window attention). @Ying1123
• Linear layers refactor.
• Make vLLM an optional dependency. @zhyncs @ByronHsu @yizhang2077 [Feature] Make vLLM optional in model code #1673

[fengyang95]

Are there any plans to optimize long context latency?

[lumiere-ml]

Hi，can I help for Multi-layer radix cache (GPU/CPU/Disk)？ Really insterested in that.

[tanzelin430]

Are there any plans to optimize long context latency?

I am interested in contributing to P-D split inference architechure and I have machines that support me to develop the architechure, if you guys got any related develop plans please let me know. Thank you @Ying1123 @zhyncs @fengyang95

[merrymercy]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

[zhyncs]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

@lumiere-ml @tanzelin430 Welcome to join our slack channel https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw

[tanzelin430]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

@lumiere-ml @tanzelin430 Welcome to join our slack channel https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw

thanks for invitation, I am in slack now. forward to collaberate with you

[lumiere-ml]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

@lumiere-ml @tanzelin430 Welcome to join our slack channel https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw

Thanks for your invitation！