Repository navigation
Conversation
satroutr
marked this pull request as ready for review
October 15, 2026 07:47
The global http_config.write_timeout is enforced as a per-connection http.Server.WriteTimeout. On a long-lived, reused HTTP/2 connection carrying back-to-back blocking queries (for example the built-in connect proxy's connect_roots watch), the in-flight stream is reset with INTERNAL_ERROR once write_timeout elapses from connection start, regardless of how long the current request has actually run. Raising the default to 15m (hashicorp#23267) only changes the reset cadence; it does not eliminate it for a persistently reused connection. Set the write deadline per blocking-query request (identified by the ?index param) based on max_query_time plus jitter and an rpc_hold_timeout buffer, never shorter than the configured write_timeout, so a long-poll is bounded by its own duration. Non-blocking requests keep the global deadline for slow-drain protection, preserving the Slowloris hardening from hashicorp#22739. Fixes hashicorp#24041 Signed-off-by: satroutr@cisco.com <satroutr@cisco.com>
satroutr
force-pushed
the
fix/blocking-query-write-deadline
branch
from
October 15, 2026 09:29
d31f4c7 to
d1ad091
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Set a per-request write deadline for blocking queries so a long-lived, reused HTTP/2 connection is not torn down mid-watch by the global per-connection
http.Server.WriteTimeout.Fixes #24041.
Why
http_config.write_timeoutis enforced as a global, per-connectionhttp.Server.WriteTimeout. On a reused HTTP/2 connection carrying back-to-back blocking queries (for example the built-in connect proxy'sconnect_rootswatch), the in-flight stream is reset withINTERNAL_ERRORoncewrite_timeoutelapses from connection start — regardless of how long the current request has run. #23267 raised the default to 15m, which only changes the reset cadence; it does not eliminate it for a persistently reused connection. Full root-cause analysis and reproduction are in #24041.How
In
HTTPHandlers.wrap, blocking queries (identified by the?indexparam, perparseWait) get a per-request write deadline viahttp.NewResponseController(resp).SetWriteDeadline(...), bounded bymax_query_time(or the request's shorterwait) plus jitter and anrpc_hold_timeoutbuffer, never shorter than the configuredwrite_timeout. Non-blocking requests keep the global deadline, preserving the Slowloris protections from #22739 (read_header_timeout,idle_timeout).Testing
go build ./agent/andgo vet ./agent/pass locally (including the new test file).TestHTTPWriteTimeout_BlockingQueryNotCutByGlobalTimeout: withwrite_timeout = "2s", a blocking query on the current index withwait=4sreturns valid JSON instead of being reset.go test ./agent/ -run TestHTTPWriteTimeoutrun and maintainer review of the design are still pending — which is why this is opened as a draft.