feat(server): OpenAI-compatible logprobs for chat and legacy completions

[Artemowka22]

Summary

Implements logprobs for sampled tokens across the whole pipeline; today /v1/completions rejects the field ("logprobs is not supported") and /v1/chat/completions silently swallows it (extra="allow").

• chat: logprobs: true + top_logprobs: 0..20 → each choice carries logprobs.content[] entries {token, logprob, bytes, top_logprobs[]}, streaming and non-streaming.
• completions (legacy): logprobs: 0..5 → {tokens, token_logprobs, top_logprobs, text_offset}. echo together with logprobs stays rejected (prompt logprobs need prefill logits and are out of scope here).

Design

• Reported values are raw model logprobs: log_softmax over the pre-temperature logits, so temperature/top-k/top-p do not change what is reported (matches vLLM's default) and greedy eval harnesses get the true model distribution.
• Zero cost when off: the sampler computes nothing unless some request in the batch asked; per-step cost when on is one log_softmax + topk + a small D2H copy that rides the existing copy_done_event.
• Engine → scheduler → detokenizer → API plumbing via optional fields (DetokenizeMsg.chosen_logprob/top_ids/top_logprobs, UserReply.logprobs), all defaulting to None — wire-compatible with older peers.
• Token strings for top-k alternatives come from the detokenizer worker (tokenizer.decode([id])); the bytes field carries UTF-8 so clients can reassemble partial-UTF-8 pieces, same trade-off OpenAI documents.
• Stop-string trimming can drop the visible text of final tokens; logprob entries still cover every sampled token.

Why

Any evaluation gate worth trusting (teacher-forced agreement/KL against a reference checkpoint, perplexity tracking of quantized variants) needs token logprobs from the OpenAI endpoint; with the radix prefix cache, per-position 1-token continuation calls make teacher-forced scoring practical without echo support.

Test plan

• CPU-only unit tests for the sampler math (tests/engine/test_sample_logprobs.py) and the entry builder (tests/tokenizer/test_logprobs_entry.py)
• FakeState API tests for both endpoints, streaming and not, plus the validation matrix (tests/server/test_logprobs_api.py) — no GPU, no weights, no network
• full server/engine/tokenizer suites pass

[HaileyStorm]

Fresh compatibility check from current main (58f4b9ec0e166205c4dfd0c6ec184ea83b5957e6): git merge-tree --write-tree origin/main origin/pr224 completed without conflicts. CPU-only focused tests on PR head ea7ae6d6d718ecd2c648474145aab8cf72f5148a passed: tests/engine/test_sample_logprobs.py, tests/server/test_logprobs_api.py, and tests/tokenizer/test_logprobs_entry.py — 12 passed in 3.63s. I was about to build the same surface for Flash-Next/ECS, found this PR during duplicate checking, and am avoiding a competing implementation. The raw pre-temperature semantics and no-work-when-disabled boundary are exactly the useful general primitive.

[HaileyStorm]

RTX 5090 / torch 2.11 cu130 microbenchmark on Qwen3.8 vocab size 248,320: the PR implementation shape (log_softmax(float logits) + chosen gather + top-20) measured 97.7 us/call at BS1, 109.1 us BS2, 115.8 us BS4 (30 warmups, 300 timed iterations, CUDA events). A split topk(logits) + logsumexp variant was not better at BS1/2 (112.6/113.6 us for top-20; 112.0 us at BS4). So the current straightforward implementation is a reasonable fast default; no speculative kernel rewrite recommended without end-to-end evidence.

[HaileyStorm]

One correctness issue before merge: streaming semantic parsing currently misattributes hidden-token logprobs to later visible content. In _generate_events_impl, every ack.logprobs is appended to pending_logprobs, but _content_delta() drains the entire pending list only when visible content is emitted. test_reasoning_logprob_is_carried_to_next_content_delta explicitly expects the logprob for thought inside <think>...</think> to be attached to the later visible answer delta. That breaks token/content alignment, can expose hidden reasoning token strings, and differs from non-streaming (which retains every sampled-token entry) and from the end-of-stream path (which drops undrained entries). I recommend fail-closing chat logprobs when semantic reasoning/tool parsing can hide/reclassify tokens, or adding token-aware routing so each public logprob entry is emitted only with its corresponding public content token. Please do not carry hidden entries onto the next visible delta. The raw legacy-completions path does not have this semantic-layer ambiguity.

[Artemowka22]

Confirmed and fixed — thanks for the precise diagnosis (and for the perf check and for deferring your own implementation).

You were right on all three counts: entries were appended before the reasoning parser classified the ack's text, _content_delta() drained the whole pending list into the next visible chunk, and test_reasoning_logprob_is_carried_to_next_content_delta enshrined exactly that behavior. The new commit closes it at both layers, taking your fail-close option:

• /v1/chat/completions now rejects logprobs: true with a 400 up front when the server runs a reasoning parser or the request enables tool parsing (tool_choice: "none" keeps logprobs available; the raw /v1/completions path is untouched).
• _generate_events_impl collects entries only on the passthrough path (no reasoning parser, no tool parsing), so a hidden-token entry can never ride a later visible delta even for a caller that skips API validation.

The carry test is replaced by a no-carry guard test plus fail-close coverage for both conflict cases and the tool_choice: "none" pass-through.

Exact token-aware routing (each public entry emitted only with its own public content token) would need the reasoning-parser interface to report per-token consumption; happy to do that as a follow-up if maintainers want logprobs and reasoning to coexist on chat.

[gdevenyi]

Tried on 2 x RTX 6000 Ada (sm_89) serving Qwen3.8-Flash-Next (RadixArk NVFP4) at TP=2, offload backend, fp8 KV pool of 8 x 262,144 tokens, merged onto my deploy branch (main af71ba4 + #385/#386/#389/#392/#354 and ten other open PRs), tests run on the box, then put in production.

Merged with conflicts against #231 (engine.py), #222 (openai_api.py) and #266 (generation.py, the content deltas now go through _content_delta after the marker filter). Findings:

• /v1/chat/completions with logprobs: true answers 400 here: logprobs on /v1/chat/completions are not supported when the server runs a reasoning parser. Every Qwen3.8 deployment runs one, so on this model only the legacy /v1/completions logprobs are usable (they work: tokens, token_logprobs, top_logprobs).
• The scheduler now reads ForwardOutput attributes, which breaks the tuple fake in tests/scheduler/test_abort_inflight_prefill.py (3 tests, 'tuple' object has no attribute 'next_tokens_cpu'); I changed the fake to build a ForwardOutput.
• tests/engine/test_sample_logprobs.py 5 passed on the GPU; serving numbers unchanged with logprobs off.