Add SGLang and NVIDIA Dynamo provider-conformance recordings

[franciscojavierarceo]

Summary

Add SGLang and NVIDIA Dynamo to an inference-provider conformance program built around versioned recordings, deterministic replay in pull-request CI, and live engine integration jobs where suitable runners are available.

The goal is to make OpenAI-compatible provider support explicit and testable without coupling the core executor to any one inference engine.

Motivation

There is interest from the llm-d community in using Agentic API with SGLang and NVIDIA Dynamo. The server already targets an OpenAI-compatible /v1/responses upstream, but compatibility currently depends too heavily on the behavior exercised by vLLM recordings.

Provider implementations differ in request-field support, event ordering, optional fields, sequence-number behavior, tool-call shapes, and terminal response envelopes. We should capture those differences as fixtures and continuously verify the gateway's typed normalization boundary.

One known SGLang compatibility case to cover is streaming events that use sequence_number: -1; those must not cause the entire event to be discarded.

Proposed design

1. Record provider traffic

• Extend the existing cassette recorder workflow rather than hand-authoring captured YAML.
• Store sanitized recordings by provider, pinned provider version, model, transport, and scenario.
• Capture the request plus the raw HTTP JSON, HTTP SSE, or WebSocket response frames before normalization.
• Record enough provenance to reproduce a cassette: engine/version, model, launch configuration, transport, and supported capabilities.
• Keep secrets, host-specific URLs, nondeterministic identifiers, and timestamps normalized or redacted.

Initial providers:

• vLLM (existing baseline)
• SGLang
• NVIDIA Dynamo, including a Dynamo deployment backed by SGLang where practical

2. Replay recordings in normal CI

• Run every provider cassette through the same request parsing, event normalization, accumulation, tool-routing, and terminal-response assertions.
• Keep replay deterministic and GPU-free so it can run on every pull request.
• Report failures with the provider, pinned version, scenario, turn, and offending frame.
• Make it easy to add a new provider or version without duplicating the test harness.

3. Add a live engine matrix

• Add scheduled and/or manually triggered integration jobs that launch pinned engine versions on appropriate runners and execute the same scenario definitions used by the recorder.
• Treat live jobs as compatibility and cassette-refresh checks; keep pull-request correctness gated on deterministic replay unless reliable GPU runners are available.
• Detect recording drift and require an intentional recorder-based cassette update.

4. Cover a shared scenario matrix

At minimum:

• blocking text response
• streaming text response
• reasoning events and summaries
• function tool call and tool output continuation
• multiple/parallel tool calls
• structured text output
• multi-turn stateful response
• HTTP SSE and WebSocket transports where supported
• incomplete, failed, malformed, and provider-specific events

Provider capability differences should be declared explicitly so an unsupported scenario is distinguishable from a regression.

Acceptance criteria

• SGLang and Dynamo recordings are committed through a documented recorder workflow.
• Provider provenance and capability metadata are stored with each recording set.
• Existing vLLM, new SGLang, and new Dynamo recordings run through one provider-neutral integration-test harness.
• Deterministic replay runs in pull-request CI.
• A pinned live-engine matrix runs on an appropriate scheduled/manual CI path.
• SGLang negative or sentinel sequence numbers are normalized without dropping otherwise valid events.
• Failures identify the provider/version/scenario/frame that diverged.
• Contributor documentation explains how to record, sanitize, replay, and update provider fixtures.

Non-goals

• Requiring byte-for-byte generated text parity across engines.
• Turning conformance tests into performance benchmarks.
• Building production engine lifecycle management into Agentic API.
• Claiming support for capabilities an engine does not expose; those should be represented in capability metadata.