Add hydrate and persist endpoints. - #216
Conversation
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
| .route("/health", get(handler::health)) | ||
| .route("/ready", get(handler::ready)) | ||
| .route("/internal/hydrate", post(handler::internal_hydrate)) | ||
| .route("/internal/persist", post(handler::internal_persist)) |
There was a problem hiding this comment.
these routes have no workload authentication or object-level authorization. once this listener is reachable by the coordinator, any reachable workload with a known response ID can hydrate its full stored history or write a turn. we should authenticate both endpoints and scope every read and write to the authenticated tenant.
| #[derive(Debug, Clone, Serialize, Deserialize)] | ||
| pub struct SplitContext { | ||
| /// Response id reserved for this turn (`resp_` prefix). | ||
| pub response_id: String, |
There was a problem hiding this comment.
SplitContext is accepted as caller-authored authority: persist never proves it came from hydrate and directly trusts its response_id, original request, continuation link, and effective configuration. a caller can skip hydrate and forge durable records; an empty response_id also returns success because persistence silently skips the write. we should use a signed, expiring context or a one-time server-side handle, and reject malformed reserved IDs.
| exec_ctx: &ExecutionContext, | ||
| ) -> ExecutorResult<ResponsePayload> { | ||
| let ctx = RequestContext::from(context); | ||
| let payload = payload_from_upstream(&ctx, upstream)?; |
There was a problem hiding this comment.
at the new HTTP persistence boundary, a response containing only {"id":"x"} defaults to Completed with empty output, while malformed output entries are silently dropped. this lets an invalid model response persist a completed turn with no valid output. we should deserialize a strict terminal response here and reject missing status, missing output, and malformed output items.
| ))); | ||
| } | ||
|
|
||
| persist_if_needed( |
There was a problem hiding this comment.
this insert makes /internal/persist non-idempotent. if the transaction commits but the response is lost, retrying the exact request hits the same primary key and returns HTTP 500 even though the turn was stored. we should use the reserved response ID as an idempotency key: return the existing result for an identical retry and 409 for mismatched reuse.
|
|
||
| // The in-process flow silently skips non-terminal statuses; here that would | ||
| // return an envelope whose id can never be continued, so reject it. | ||
| if !matches!( |
There was a problem hiding this comment.
response.failed is a terminal upstream result, but it becomes ResponseStatus::Error and is rejected here as InvalidRequest, so /internal/persist reports a model failure as HTTP 400. we should return the normalized failed response or a dedicated upstream-error status, and reserve 400 for malformed or non-terminal input.
| let shutdown = CancellationToken::new(); | ||
| let on_signal = shutdown.clone(); | ||
| tokio::spawn(async move { | ||
| if tokio::signal::ctrl_c().await.is_ok() { |
There was a problem hiding this comment.
this only handles Ctrl-C, while Kubernetes terminates containers with SIGTERM. the default SIGTERM action bypasses the graceful-shutdown future and can cut off in-flight hydrate or persist requests. we should reuse the gateway's SIGTERM-aware shutdown path and bounded drain timeout.
2 participants
Summary
Exposes the two halves of a stateful Responses turn as separate endpoints, so an external orchestrator (the llm-d coordinator) can make the inference call itself:
POST /internal/hydrate: expandsprevious_response_idinto a stateless upstream request, plus an opaque context.POST /internal/persist: takes that context and the model's response (a JSON body, or the SSE frames a streaming caller relayed), stores the turn, and returns the envelope carrying the storedresp_id.Both compose existing in-process steps (
rehydrate_conversation,payload_from_upstream,persist_if_needed). Sosplit.rsadds no parsing, storage, or request building of its own.RequestContextis unchanged for in-process callers; it now serializes through a reduced wire form,SplitContext.The endpoints ship as a new crate and binary,
agentic-llm-d, depending onagentic-server-corealone.Test Plan
cargo test- newsplit_execution_integration.rscovers the two-turn replay, the error paths, and the context round-trip. Existing suites pass unchanged.Verified on a cluster with a multi-turn conversation.
Closes #215.