persist: record the parameter-tuning discussions in the design doc

Capture the warm-interval analysis (the purge-window mechanism, why
abort-and-restart warming probes better than a patient handshake, and
why the interval stays at 20 seconds pending evidence), the tuning
trade-offs of the remaining parameters (the delay's rescue floor vs
its false-fire population, the bucket-capacity sizing rationale, the
dead-socket slot-pinning interaction), and open Alternatives with the
naive options (the primary's own pool, just lowering the timeouts),
each alternative in its own subsection. Calibrate the pool-isolation
motivation to the evidence along the way, in the doc and the hedge.rs
module doc: same-pool traffic mostly survived the observed events, so
the case for isolation is the correlated residual plus what a shared
pool forecloses, not blanket fate sharing. Also retire the informal
"retry ladder" term in favor of naming retry_external.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: ggevay

doc/developer/design/20260806_hedged_blob_gets.md

@@ -13,7 +13,8 @@ succeeds first.
 - Cap the freshness impact of the dead-connection failure class (described
   below) at roughly the hedge delay plus one normal get, whenever a healthy
   connection is reachable. When it is not (both requests stall), behavior
-  falls back to today's, with the existing retry ladder as the backstop.
+  falls back to today's, with persist's existing retry-with-backoff
+  machinery (`retry_external`, described in The Problem) as the backstop.
 - Change nothing about persist's semantics, error surface, or write paths.
 
 ## Non-Goals
@@ -112,8 +113,8 @@ starts immediately and races a timer set to the configured delay. If the
 primary completes first (the full blob, not just its first byte: `Blob::get`
 returns a complete result, so partial progress is invisible to the
 wrapper), its result is returned verbatim, success or error.
-A fast error therefore still takes today's path into the `retry_external`
-ladder. Hedging targets hangs, not failures.
+A fast error therefore still takes today's path into `retry_external`'s
+retries. Hedging targets hangs, not failures.
 
 If the delay elapses, the hedge leg fires on the sibling, subject to the
 two admission guards described under Bounding amplification below. If the
@@ -128,7 +129,7 @@ error. The two error cases are asymmetric. If the hedge leg errors, the
 get keeps waiting on the primary alone. If the primary errors after the
 hedge fired, the get waits a delay-sized grace window for the hedge and
 then returns the primary's error, so a slow hedge cannot hold the get past
-the point where handing the error to the retry ladder is the better move.
+the point where handing the error to `retry_external` is the better move.
 Either way, when the get does fail, the caller sees the primary's error
 object, unchanged. This matters because `ExternalError::is_timeout`
 matches on the error string, so even attaching the hedge's error as context
@@ -170,9 +171,10 @@ timer that fires late is a hedge that does not fire.
 
 ### Pool isolation
 
-The hedge is only useful if it cannot be assigned the same dying
-connection, so the sibling must not share connection state with the
-primary. Three facts make that work.
+The sibling shares no connection state with the primary, so a hedge
+cannot be handed a connection dying in the same event that stalled the
+primary. The full case for isolation, including what a shared pool
+forecloses, is under Alternatives. Three facts make the isolation work.
 
 First, isolation cannot come from cloning: the AWS SDK client embedded in
 `S3BlobConfig` is reference-counted, so cloning the config shares the HTTP
@@ -230,6 +232,40 @@ counter and leaves the gauge alone, so a fast-failing sibling (for
 example, one whose credentials rotted) cannot masquerade as a fast healthy
 one.
 
+Because the killed ping closes its socket, the warm interval also bounds
+how long a dead sibling socket can linger in the pool undetected: about
+two intervals (up to one until a ping lands on it, one more until the
+cycle timeout kills it). That window is what decides whether hedging
+survives a correlated event that hits the sibling's own sockets. Until
+the purge, a hedge can check out the dead socket and hang until the read
+timeout while contributing nothing, so at the current interval hedging
+protects against such an event only if the sibling happened to escape
+it. Such a hedge also holds its concurrency slot for the whole hang, so
+the event's first hedges can pin both default slots for up to the read
+timeout, leaving later gets unhedged with
+`hedges_skipped{reason="concurrency"}` climbing. A single-digit interval would shrink the exposure to a few seconds
+and let the pool heal within the event itself. The healing loop is also
+a better prober than one patient handshake: each restarted attempt
+resets TCP's exponential SYN backoff, probing the path about once a
+second instead of ever more sparsely, and performs a fresh DNS
+resolution, which can retarget a rotated S3 front-end address once the
+record's short TTL lapses. The 2023-era caution against aggressive
+connect timeouts does not apply here, because a warming handshake has no
+request waiting on it, so aborting a viable-but-slow attempt costs
+nothing.
+
+The interval nevertheless stays at 20 seconds, as a cost decision. At 20
+seconds the warmer's requests are a modest fraction of the fleet's
+organic blob gets and their request cost is negligible. At a few seconds
+the warmer would rival or exceed the fleet's entire organic blob-get
+request volume, a spend increase that deserves a deliberate decision
+backed by evidence rather than a default. If enabled hedging still
+leaves correlated events visible in freshness, with hedges erroring on
+dead sibling sockets, concurrency skips clustering at event times, and
+the warm-error counter spiking alongside, that is the evidence that the
+interval is too coarse, and it is a dyncfg, so it can be lowered at
+runtime without a release.
+
 The warmer runs only while hedging is enabled, re-checking the flag at its
 normal cadence while idle. While hedging is disabled the sibling is
 therefore fully idle: no requests, no credential refreshes (SDK providers
@@ -263,7 +299,11 @@ request rate onto an already-degraded dependency. The same guard keeps
 large gets that legitimately exceed the delay (a 128 MiB part on a
 bandwidth-constrained pod) from settling into permanent double egress.
 The bucket starts full so that low-traffic processes can still hedge the
-rare event that motivates the feature. The known blind spot is the mirror
+rare event that motivates the feature. The capacity is also sized
+against the events themselves: the observed damage mechanism is a few
+hung gets on hot paths starving everything downstream, so one event
+needs a handful of rescues per process, and a full bucket of 32 is an
+order of magnitude above that. The known blind spot is the mirror
 of the protection: in a window where more than one percent of gets are
 legitimately slow, the drained bucket will also refuse the occasional
 genuine dead-connection hang, visible as `hedges_skipped{reason="budget"}`.
@@ -285,6 +325,20 @@ than an operational lever, and fewer knobs means less to review and fewer
 LaunchDarkly entries. Note that `budget_ratio = 0` is not a kill switch,
 because the bucket starts full. `enabled` is the kill switch for hedging.
 
+The delay is the knob most tempting to retune, so the trade is worth
+recording. Lowering it moves the rescue floor (a hedge win arrives no
+earlier than the delay plus one round trip) but grows the false-fire
+population: the race triggers on full-blob completion, not first byte,
+so the gets that legitimately outlast the delay are large parts on
+constrained bandwidth, and each false fire both duplicates that part's
+egress and spends a budget token, so a lower delay can drain the bucket
+with healthy traffic and leave a real event refused. Raising it has a
+wide plateau in the other direction: a rescue stays useful as long as
+the delay plus one normal get fits the freshness target, which holds up
+to several seconds. Any retune should start by re-measuring the
+would-be fire rate at the candidate delay (the rate in Rollout is
+workload-dependent, dominated by part sizes and pod bandwidth).
+
 Honesty about what the kill switch covers: `enabled = false` stops both
 hedge requests and the warmer, leaving the sibling fully idle, but the
 sibling client and its credential chain are constructed at process start
@@ -395,9 +449,44 @@ process start.
 
 ## Alternatives
 
-Lowering `persist_blob_read_timeout` does not cover the class: as The
-Problem notes, it is first-byte-only, and the characterized connections
-died before any timeout fired at all.
+### Hedging on the primary's pool
+
+Hedging on the primary's own connection pool would need no sibling
+machinery at all, and in the dominant event shape (one connection dies
+while its pool-mates stay healthy, which is why the same-pool retry
+succeeds promptly today) it would usually work. The case for isolation
+is the residual minority of correlated, path-scoped events, where we
+observed kills clustering within a second on one pod and fresh
+connects stalling fleet-wide: there, a shared pool is most likely to
+hand the hedge a sick connection exactly when the hedge matters most.
+A shared pool also forecloses levers that matter regardless of fate
+sharing: it pins the hedge to the pool's cached DNS answer and
+front-end address, lets a checkout queue behind the very jam the hung
+get is part of, and cannot be kept warm without distorting the
+primary's pool. Isolation makes the hedge's independence structural in
+precisely the worst events, and the mid-event evidence in The Problem
+(most collections on the same pod stayed fast while one hung) shows
+independent healthy paths exist to be reached.
+
+### Just lowering the client timeouts
+
+Lowering the client timeouts instead of hedging was the 2023 answer to
+this class (the connect and read timeouts were cut from 30 and 60
+seconds to today's 7 and 10), and the residual hang is what that lever
+left behind. Tightening further runs into structural limits. The read
+timeout is first-byte-only and the characterized connections died
+before any timeout fired at all, and a mid-body hang is bounded only by
+the 90 second attempt timeout, which cannot come down to freshness
+timescales without killing every legitimately long large-part fetch. A
+timeout is also a sequential remedy with a forced trade: it converts a
+possibly-about-to-succeed request into an error, pays the backoff and
+the restart, and retries on the same pool with no fresh connection or
+DNS resolution guaranteed, so its threshold must stay conservative. A
+hedge is concurrent: the primary keeps running, so firing early costs a
+bounded duplicate rather than a lost success, which is why hedging can
+afford a 2 second trigger where a 2 second timeout could not.
+
+### Per-chunk hedging
 
 Hedging per multipart chunk inside `S3Blob::get` is a possible refinement:
 `S3Blob::get` already fetches large blobs in 8 MiB parts, so a per-chunk
@@ -406,21 +495,29 @@ on missing first bytes rather than wall clock. It is S3-only, duplicates
 the race logic per backend, and is a natural follow-up once the generic
 version has proven itself, not the first version.
 
+### A pool-less sibling
+
 Disabling pooling on the sibling client (max idle connections of zero)
 would guarantee isolation and fresh DNS with no warmer machinery, but it
 puts a full TCP and TLS handshake on the critical path of every hedge,
 which is up to 7 seconds during exactly the correlated events the feature
 targets.
 
+### A both-legs debug mode
+
 A debug mode that awaits both legs and asserts they agree was rejected as
 unsound, not merely unnecessary: as Correctness explains, two gets of the
 same key are only required to agree for keys protected by a live lease, so
 the assertion would misfire on legitimate delete races.
 
+### A hedging-aware Blob trait
+
 Extending the `Blob` trait with a hedging-aware method was rejected because
 it pushes a transport concern into a six-method correctness-critical trait
 that most implementations have no answer for.
 
+### Hedging in maelstrom
+
 A configuration for maelstrom (persist's Jepsen-style linearizability test
 harness) was considered and deferred. Maelstrom's blob is neither S3 nor
 Azure, so its sibling would be the same instance, and `UnreliableBlob`

src/persist/src/hedge.rs

@@ -25,9 +25,10 @@
 //! streaming interface. Extending hedging to any of them is forbidden.
 //!
 //! The hedge handle must not share a connection pool (or DNS state) with the
-//! primary, otherwise the hedge can be assigned the very connection that is
-//! dying. See [crate::cfg::open_hedge_sibling] for how that isolation is
-//! constructed per backend.
+//! primary, otherwise the hedge can be handed a connection dying in the same
+//! event that stalled the primary, exactly when a hedge matters most. See
+//! [crate::cfg::open_hedge_sibling] for how that isolation is constructed
+//! per backend.
 //!
 //! Hedging operates within a single `retry_external` attempt, before any
 //! failure surfaces. The retrying in `retry_external`, which is what