A conformance suite and open dataset for LLM tool-call translation.
Every LLM gateway, proxy, and OpenAI-compatible server translates between wire dialects: OpenAI Chat Completions, Anthropic Messages, and the OpenAI Responses API. Translation is lossy by construction, because each dialect carries fields the others do not. The question is what a gateway does with the fields it cannot carry. The correct answers are to map them, to refuse the request, or to report the loss. The common answer is to drop the field and return HTTP 200.
Tool calls are where this matters. An agent loop does not read the model's
text to decide what to do next. It reads the stop reason, the tool-call id, and
the arguments. When a gateway relabels tool_use as end_turn, rewrites an id
without an inverse, or stringifies an image inside a tool result, the loop
halts, races, or goes blind, and the model gets blamed.
kairo reproduces these failures on the wire, freezes each one as a deterministic replay test, and scores any translation layer against the resulting invariants. Every finding is backed by recorded bytes, a control that succeeded on the same input, and an N of N reproduction count. The suite runs offline with no provider keys.
| Metric | Value |
|---|---|
| Reproduced issue folders | 52 |
| Gateways under test | LiteLLM, NVIDIA Switchyard, Bifrost, GoModel, AxonHub, and any-llm |
| Harness tests | 161 (134 conformance checks against recorded transcripts, 27 unit) |
The 52 folders cover reproduced findings, multi-defect reports, and honest
negative results. Versions and reproduction outcomes are recorded per finding
in issues/SCOREBOARD.md, including cited bugs that did
not reproduce.
One finding is filed upstream as
NVIDIA-NeMo/Switchyard#380.
Independent gateways violate the same small set of invariants, and a checker written against one gateway's transcript catches the same defect in the others without modification. That is the central result. The table groups every reproduced defect by the invariant it violates. Each number links to the folder with the writeup, the bytes, and the reproduction commands.
| Invariant | LiteLLM | Switchyard | Bifrost | GoModel | AxonHub | any-llm |
|---|---|---|---|---|---|---|
Terminal reason survives translation (tool_use, content_filter, max_tokens, refusal) |
001, 002 | 010 | 030, 034, 035, 036 | |||
| One public stream has one response lifecycle and output namespace | 074 | |||||
Request constraints survive (disable_parallel_tool_use, stop_sequences, tools[].strict, output_format) |
017, 041, 064 | 006, 017, 040, 065, 066 | 031, 032, 072 | 042, 043 | 051 | 058, 062 |
Content blocks survive (refusal, is_error, image and document blocks in tool results and user turns) |
006, 007, 018, 067 | 006, 007, 018, 068, 069 | 059, 060, 061 | |||
Assistant history survives replay (thinking blocks and signatures) |
016 (leaked as visible text) | 016 (dropped) | 033 | 057 | ||
| Tool-call ids round-trip | 004 | 005 | 037 | |||
Nothing is invented (empty text blocks, phantom message items, cache_control) |
001, 009 | 019, 045, 068 | ||||
| Malformed input fails closed | 008 | |||||
| Client credentials stay client-side | 020, 024, 026, 028, 071 | 023, 025, 027, 063 |
The disable_parallel_tool_use flag is dropped by five of the six gateways.
The Anthropic {"type": "auto", "disable_parallel_tool_use": true} object
becomes the bare string "auto", and no parallel_tool_calls: false appears
on the OpenAI-shaped side. The
017 checker caught Switchyard and LiteLLM
first and then Bifrost, GoModel, and any-llm unchanged.
Two gateways route Anthropic /v1/messages through the OpenAI Responses API
rather than Chat Completions, even when the configured backend is openai/*.
Field names change again on that hop (messages becomes input, system
becomes instructions, max_tokens becomes max_output_tokens). A probe
corpus written against Chat Completions spellings scores those fields as
dropped when they were carried. Twelve cells in an early sweep were false
drops for that reason, and the corpus now checks both spellings.
The same field fails in different ways across gateways, and the failure mode
matters. Replayed thinking blocks are dropped by Switchyard, Bifrost, and
any-llm, which breaks reasoning continuity and prompt caching. LiteLLM instead
forwards them as visible output_text, which puts private reasoning into the
model's visible context. An image inside a tool_result is JSON-dumped into a
text string by Switchyard, so the model receives literal base64, and is deleted
outright by LiteLLM.
Honest negatives are kept as data. Bifrost's multimodal handling and its handling of client credentials are correct where both incumbents fail. Switchyard's streaming tool-call re-encoder reassembles split argument deltas correctly. Several cited upstream tickets are patched on current releases and are recorded as non-reproductions. Issue 030 is a regression of a Bifrost bug fixed in v1.5.4, which is the argument for a permanent suite rather than a one-time audit.
Every accepted finding has three legs.
- Wire evidence. The bytes as sent and received, under
transcripts/. Raw SSE or JSON. Screenshots and paraphrases are not evidence. - A control that works. The same input succeeding somewhere: the model called directly, another route on the same gateway, or another version. The control isolates the translation layer as the cause and rules out the model and the prompt.
- Determinism. N of N reproductions, with the trigger narrowed to the exact field, length, or chunk shape. A failure that reproduces only sometimes is a lead and needs narrowing before it becomes a finding.
A finding becomes a checker in
crates/harness/src/checks.rs that encodes the
invariant rather than the bug:
pub enum Verdict {
Conformant,
Violation(String),
}
/// If a streamed Anthropic response contains a `tool_use` block, the
/// terminal `stop_reason` MUST be `tool_use`.
pub fn anthropic_toolcall_stop_reason(sse: &str) -> VerdictA test in
crates/harness/tests/conformance.rs
asserts the verdict against the recorded bytes. Run against a buggy gateway's
transcript, the checker returns the violation. Run against a correct
implementation, it returns conformance. The same checker scores both
directions, which is what lets the suite grade a target instead of arguing
about it. A Violation assertion is a frozen bug: the day a gateway stops
violating the invariant, the test flips and says so.
client dialect backend dialect
┌────────────┐ Anthropic Messages ┌──────────┐ Chat / Responses ┌───────────────┐
│ agent or │ ───────────────────> │ gateway │ ─────────────────> │ capture mock │
│ curl │ <─────────────────── │ under │ <───────────────── │ or live model │
└────────────┘ SSE / JSON │ test │ canned or real └───────────────┘
│ └──────────┘ │
│ response bytes forwarded request │
└────────────────> transcripts/NNN/ <────────────────────────────────┘
│
v
crates/harness (checkers + conformance tests)
Two capture rigs cover the findings.
The offline capture rig points the gateway's backend at
tools/mock_upstream.py, which appends every
forwarded request body to a JSONL file and replies with a canned SSE stream or
JSON body. This exposes encode-side losses, meaning fields that were in the
client request and are absent from what reached the backend. It needs no keys
and is fully deterministic. Most request-constraint, tool-result, history, and
credential findings were captured this way.
The live capture rig runs the gateway against a real backend (Ollama, Gemini, Kimi, Anthropic, OpenAI) and records both directions. This exposes decode-side failures, meaning stop reasons, ids, and content the gateway produced for the client, and it measures what a loss costs a real run.
The harness is a single Rust crate with no network access. Checkers operate on
raw SSE bodies, response JSON, and capture JSONL. The workspace forbids unsafe
code and denies all clippy lints at the pedantic level. CI runs the README
counter check, rustfmt, clippy, unit, conformance, and doc tests, and
rustdoc with warnings denied.
crates/harness/src/checks.rs invariant checkers, one per defect class
crates/harness/tests/conformance.rs one test per recorded transcript
issues/NNN-slug/README.md writeup: what breaks, evidence, control, invariants, repro
issues/SCOREBOARD.md every finding, version, ticket, and result
issues/MATRIX.md field-preservation matrix from the sweep rig
issues/TARGETS.md unclaimed upstream tickets to reproduce
transcripts/NNN/ recorded bytes per finding
tools/mock_upstream.py offline capture backend
tools/capture_server*.py request recorders for specific dialects
tools/sweep/ rectangular gateway x probe sweep
tools/update-readme-counts.py regenerates the Status block; CI fails if stale
| Axis | Covered |
|---|---|
| Client dialects | Anthropic Messages, OpenAI Chat Completions, OpenAI Responses |
| Backend dialects | OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, Gemini, Ollama |
| Gateways | LiteLLM, NVIDIA Switchyard, Bifrost, GoModel, AxonHub, any-llm |
| Modes | streaming and non-streaming, per route |
| Surfaces | stop and finish reasons, tool-call ids, argument assembly, request constraints, multimodal tool results, replayed history, invented fields, error handling, credential handling |
Versions are pinned in each writeup. Findings are stated against the exact release or commit they were captured on, and the scoreboard records whether a cited bug reproduces on the current release.
Replay the suite. No keys are needed because the tests read recorded bytes.
cargo testReproduce a finding offline. This is issue 017 against Switchyard: the
forwarded tool_choice arrives as the string "auto" with no
parallel_tool_calls field.
python tools/capture_server.py $PWD/transcripts/016/cap-parallel.jsonl &
tools/switchyard/target/release/switchyard-server --config tools/switchyard-capture.toml --port 9000 &
curl -s localhost:9000/v1/messages -H 'anthropic-version: 2023-06-01' \
-d @transcripts/016/req-parallel.jsonReproduce a finding live. Keys are used for capture only and are never committed.
cp .env.example .env
# add provider keys, then follow the repro block in any issues/NNN/README.mdEach folder under issues/ follows issues/TEMPLATE.md
and states, in order: the upstream ticket and its state on the reproduction
date, the tool and exact version under test, the reproduction date and
environment, what breaks and which agent loops it hurts, the wire evidence
with file names, the control matrix, the root cause if it was pinned to a
source line, and the invariants the bug implies. A writeup says what was
checked and what was not. Where the reproduction path differed from the
upstream reporter's configuration, the writeup says so and explains what that
difference does to the claim.
Issue folders answer "does gateway X drop field Y". The sweep rig under
tools/sweep/ answers the question underneath: of every field a
cross-format gateway has to carry, how many survive, on each gateway, measured
the same way. It runs every gateway against every probe, repeats non-clean
cells to N runs, and writes issues/MATRIX.md with a
preservation rate per gateway and a legend that separates a dropped field from
a field with no equivalent in the target format and from a gateway that could
not be started. An absent gateway and a clean gateway never look the same in
the results.
The sweep produces leads and frozen bytes. It does not write issue folders. Every folder remains a hand-verified claim with a control and a determinism count, one bug per pull request.
If a tool call broke behind a gateway, the transcript is the contribution. No diagnosis is required.
Install the /kairo-report command into Claude Code once:
mkdir -p ~/.claude/commands && curl -fsSL https://raw.githubusercontent.com/Atharva-Kanherkar/kairo/main/agent-commands/claude-code/kairo-report.md -o ~/.claude/commands/kairo-report.mdThe next time a tool call fails, run /kairo-report in that session. The
agent gathers the evidence, redacts secrets, shows the report, and files it
after confirmation. The manual route is a
tool-call failure report.
New reproductions are the highest-value contribution, and proving a bug is
real is sufficient. A fix is not required. Non-reproductions of cited bugs are
recorded as data. The method, the pull-request checklist, and the style rules
are in CONTRIBUTING.md. Unclaimed targets, including
vLLM, SGLang, Ollama, and claude-code-router tickets, are listed in
issues/TARGETS.md.
kairo tests translation layers. It does not benchmark model quality, and a model that declines to call a tool is not a finding. It does not ship fixes to the gateways it tests; findings are filed upstream and linked from the writeup. It hunts silent failures first: a gateway that returns a clean 4xx for an unsupported field is recorded as loud and correct, and the checker grammar distinguishes a dropped field from a field with no equivalent in the target dialect.
The end goal is a router that lets any coding agent run on any model with tool calls that survive translation. The router will be built on this dataset and scored by this suite, and it is not started. Nearer work is widening the gateway column (vLLM, SGLang, Ollama's OpenAI compatibility layer, claude-code-router), completing the sweep across all six gateways on current releases, and filing the remaining unfiled findings upstream.
Apache-2.0. See LICENSE.