Skip to content

Add experimental server with chat completions endpoint - #353

Merged
neilmehta24 merged 18 commits into
mainfrom
neil/mlx-server
Aug 10, 2026
Merged

Add experimental server with chat completions endpoint#353
neilmehta24 merged 18 commits into
mainfrom
neil/mlx-server

Conversation

@neilmehta24

@neilmehta24 neilmehta24 commented Jul 23, 2026

Copy link
Copy Markdown
Member

Summary

This PR adds a small authenticated HTTP server for text generation. It exposes health and streaming chat-completion endpoints backed by the existing mlx-engine APIs.

Design

This is not intended to be a general OpenAI-compatible server. Its contract is intentionally limited to the initial text and vision generation path. It assumes one loaded model, a trusted local client, streaming responses, and inline base64 images. Unsupported capabilities fail explicitly rather than being approximated.

generate.py remains the source of truth for generation behavior. The server only validates and translates requests. The loaded model owns the chat template. Clients may supply template variables, but cannot replace the template or override rendering controls. Tool-enabled requests are rejected because the server does not yet emit semantic tool-call events. Images are never fetched from URLs or files.

The continuous batcher previously decoded for up to 500 ms before checking for new requests. This amplified small differences in HTTP request arrival time. It now yields when work is waiting.

Performance

Test Direct Python HTTP
Text generation 242.9 tok/s 246.2 tok/s
Vision generation 117.7 tok/s 117.2 tok/s
Uncached vision TTFT 406.0 ms 407.8 ms

Direct Python and HTTP requests used the same loaded model, prompt, and deterministic generation settings. Follow-up parity checks produced identical text, completion-token counts, and stop reasons for both text and vision generation. Qwen2.5 0.5B Instruct 4-bit was used for text generation and Gemma 4 E2B 4-bit for vision generation.

@github-actions github-actions Bot added the CLA signed Indicates that all contributors have signed label Jul 23, 2026
@neilmehta24
neilmehta24 marked this pull request as ready for review July 30, 2026 20:40
@neilmehta24
neilmehta24 merged commit 35a325f into main Aug 10, 2026
2 checks passed
@neilmehta24
neilmehta24 deleted the neil/mlx-server branch August 10, 2026 14:55
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 10, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

CLA signed Indicates that all contributors have signed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants