Skip to content

fix(grpc): release request payloads at dispatch via RequestLease - #2242

Closed
slin1237 wants to merge 1 commit into
fix/grpc-request-payload-lifetimefrom
fix/grpc-dispatch-release
Closed

fix(grpc): release request payloads at dispatch via RequestLease#2242
slin1237 wants to merge 1 commit into
fix/grpc-request-payload-lifetimefrom
fix/grpc-dispatch-release

Conversation

@slin1237

Copy link
Copy Markdown
Member

Description

Problem

With response processing on views, the parsed gRPC request still lives in the pipeline context and in the router retry loop until the response completes (non-streaming) or the response head (streaming) — even when retries are disabled and nothing can replay it.

Solution

Consume the shared RequestLease: the route implementations lease the canonical body (release point derived from the effective retry config), each attempt clones the payload handle out of the lease, and RequestExecutionStage drops the context's payload handle and releases the lease right before the send. Retries disabled frees the request at dispatch; retries enabled keeps it exactly until the retry window closes (lease drop). Harmony pipelines opt out: their tool loop re-reads the request across iterations, and gpt-oss requests carry no multimodal payloads.

Changes

  • request_lease.rs: object-safe ErasedLease for threading the lease through the heterogeneous stage context, plus SerializedBody::Sized / note_upstream_len so the typed-proto pipeline feeds the released-early metric with prost wire sizes
  • RequestInput carries the lease and a construction-time streaming flag; RequestType::Released(RequestKind) keeps post-dispatch stage dispatching payload-free
  • RequestExecutionStage releases before dispatch; ExecutionPlan::wire_len sums prost encoded_len
  • Route implementations for chat/generate/completion/messages create the lease and thread it (chat skips it for Harmony)

Test Plan

New probes on the gated stub: disabled_retries_release_parsed_request_before_upstream_responds (+ grpc_pd twin; the stub refuses the generate RPC until the request is freed) and enabled_retries_replay_identical_token_ids (induced UNAVAILABLE, identical replayed input ids, freed at lease drop). Suites: routers:: unit tests (631), routing_tests (120), tenant_rate_limiting_grpc_test (14), zmq_backend_test (13), spec_test (96) — all green.

Stacked on the request-view PR.

Checklist
  • cargo +nightly fmt passes
  • cargo clippy --all-targets --all-features -- -D warnings passes
  • (Optional) Documentation updated
  • (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

Signed-off-by: Simo Lin <25425177+slin1237@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 302c8e35-2d01-4745-a42a-9c2e882ed52d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added grpc gRPC client and router changes model-gateway Model gateway crate changes labels Aug 20, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, well-designed PR. The lease lifecycle, view extraction, and Released variant handling are all correct. Tests comprehensively cover the key invariants (streaming release, PD release, dispatch-time release with retries disabled, and replay with retries enabled). No issues found.

@slin1237

Copy link
Copy Markdown
Member Author

Superseded together with #2239 by the pipeline-ownership refactor — the lease becomes structural in the new pipeline contract rather than threaded through contexts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

grpc gRPC client and router changes model-gateway Model gateway crate changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant