Skip to content

refactor: unify response processing under AgentPipeline - #274

Merged
maralbahari merged 6 commits into
vllm-project:mainfrom
EmbeddedLLM:rfc-243
Sep 11, 2026
Merged

refactor: unify response processing under AgentPipeline#274
maralbahari merged 6 commits into
vllm-project:mainfrom
EmbeddedLLM:rfc-243

Conversation

@maralbahari

Copy link
Copy Markdown
Collaborator

Summary

The Responses executor previously split JSON/SSE accumulation, tool-call translation, and stream emission across upstream.rs, accumulator.rs, and function_sse.rs. This made ownership unclear and allowed live and collected responses to follow different processing paths.

This PR implements #243 from RFC #241 by introducing a request-owned AgentPipeline:

  • Route complete JSON and live SSE bodies through the same pipeline and finalization path.
  • Add a synchronous, round-scoped RoundIngestion that combines typed response accumulation with tool-call translation.
  • Split ResponseAccumulator into focused modules with typed active/completed slots keyed by validated output_index.
  • Apply strict and lenient validation policies through the same state transitions.
  • Replace the monolithic function SSE translator with an inline dispatcher and tool-specific translators for function, custom, namespace, and tool-search calls.
  • Keep ToolRegistry as the request-scoped lookup for available tools and their client/gateway ownership. Translators consume an owned classification snapshot.
  • Keep gateway-owned tool execution and public lifecycle generation in the gateway path.
  • Move ordered, awaited stream delivery into AgentPipeline, preserving cross-round sequencing, output-index rebasing, deferred event ordering, and downstream backpressure.
  • Encapsulate the tool loop, registry, response budget, and pipeline under EngineOrchestration.
  • Keep upstream.rs focused on request serialization, inference transport, response-budget accounting, and selecting the JSON or SSE pipeline runner.
  • Document the final ownership boundaries, canonical tool normalization, client translation, and OutputItem::to_input_item as the continuation conversion policy.
  • Refresh the gateway replay cassettes for custom tools, MCP, parallel tool calls, tool search, and web search.

Closes #243.
Part of #241.

Test Plan

  • cargo test --locked --workspace -- --test-threads=2
    • 1120 passed, 0 failed, 9 ignored
  • cargo clippy --locked --workspace --all-targets -- -D warnings
  • git diff --check
  • Re-recorded and replayed the affected gateway cassettes.

- Unify JSON and live SSE ingestion under AgentPipeline
- Move accumulation, stream delivery, and request state into the pipeline
- Add tool-specific translators with an inline dispatcher
- Move response restoration from ToolRegistry into translation
- Keep tool-search state in the pipeline and lookup validation in the registry
- Encapsulate tool-loop orchestration in EngineOrchestration
- Expand coverage for translation, streaming, and state ownership

Signed-off-by: maral <maralbahari.98@gmail.com>
Signed-off-by: maral <maralbahari.98@gmail.com>
Signed-off-by: maral <maralbahari.98@gmail.com>
Signed-off-by: maral <maralbahari.98@gmail.com>

@franciscojavierarceo franciscojavierarceo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed ingestion, streaming delivery, tool translation, and persistence boundaries at e03fb7e. No actionable regressions found. Validation: 877 core tests and 19 split-execution tests passed; 9 tests were ignored. Cassette replay coverage passed; live providers were not tested.

Signed-off-by: maral <maralbahari.98@gmail.com>
Signed-off-by: maral <maralbahari.98@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor: unify synchronous streaming ingestion and item lifecycle

2 participants