fix(grpc): release request payloads at dispatch via RequestLease - #2242
fix(grpc): release request payloads at dispatch via RequestLease#2242slin1237 wants to merge 1 commit into
Conversation
Signed-off-by: Simo Lin <25425177+slin1237@users.noreply.github.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
There was a problem hiding this comment.
Clean, well-designed PR. The lease lifecycle, view extraction, and Released variant handling are all correct. Tests comprehensively cover the key invariants (streaming release, PD release, dispatch-time release with retries disabled, and replay with retries enabled). No issues found.
|
Superseded together with #2239 by the pipeline-ownership refactor — the lease becomes structural in the new pipeline contract rather than threaded through contexts. |
Description
Problem
With response processing on views, the parsed gRPC request still lives in the pipeline context and in the router retry loop until the response completes (non-streaming) or the response head (streaming) — even when retries are disabled and nothing can replay it.
Solution
Consume the shared
RequestLease: the route implementations lease the canonical body (release point derived from the effective retry config), each attempt clones the payload handle out of the lease, andRequestExecutionStagedrops the context's payload handle and releases the lease right before the send. Retries disabled frees the request at dispatch; retries enabled keeps it exactly until the retry window closes (lease drop). Harmony pipelines opt out: their tool loop re-reads the request across iterations, and gpt-oss requests carry no multimodal payloads.Changes
request_lease.rs: object-safeErasedLeasefor threading the lease through the heterogeneous stage context, plusSerializedBody::Sized/note_upstream_lenso the typed-proto pipeline feeds the released-early metric with prost wire sizesRequestInputcarries the lease and a construction-timestreamingflag;RequestType::Released(RequestKind)keeps post-dispatch stage dispatching payload-freeRequestExecutionStagereleases before dispatch;ExecutionPlan::wire_lensums prostencoded_lenTest Plan
New probes on the gated stub:
disabled_retries_release_parsed_request_before_upstream_responds(+ grpc_pd twin; the stub refuses the generate RPC until the request is freed) andenabled_retries_replay_identical_token_ids(induced UNAVAILABLE, identical replayed input ids, freed at lease drop). Suites:routers::unit tests (631),routing_tests(120),tenant_rate_limiting_grpc_test(14),zmq_backend_test(13),spec_test(96) — all green.Stacked on the request-view PR.
Checklist
cargo +nightly fmtpassescargo clippy --all-targets --all-features -- -D warningspasses