Agent topology-refresh poll: hardcoded 10s interval, no jitter, no 429 backoff — sustained hub rate_limited under multi-replica HA

[jacaudi]

Summary

With a tunnel agent run at 2 replicas (for access HA), the towonel-agent pods get sustained HTTP 429 (rate_limited) from the hub on their periodic topology refresh (POST /v1/bootstrap):

WARN topology refresh failed; keeping current edge set
     error: hub returned 429 (rate_limited): Too Many Requests! Wait for 0s
... repeats every ~10s on BOTH pods; "Wait for" hint grows 0s -> 1s -> 3s

Non-fatal per request (keeping current edge set), but it defeats the point of 2 replicas: while throttled the agents can't refresh edge topology, so failover to a replacement edge can stall.

Surfaced by the downstream Kubernetes operator, jacaudi/towonel-operator#45. 2 agents per tunnel is a hard HA requirement there — dropping to 1 is not an acceptable fix.

Root cause (reading main)

1. Hardcoded, non-overridable interval — crates/towonel-agent/src/main.rs:

   /// How often the agent re-queries the hub for the current trusted-edge set.
   const TOPOLOGY_REFRESH_INTERVAL: Duration = Duration::from_secs(10);

   No env var / CLI flag reads it; agent config only parses the service env vars.
2. No jitter — the loop is a fixed tokio::time::sleep(TOPOLOGY_REFRESH_INTERVAL), so replicas that boot together poll in lockstep.
3. 429 wait hint ignored — crates/towonel-agent/src/hub_client.rs check_response reads only status() + bytes(); the bootstrap poll isn't wrapped in the retry layer, so a 429 just logs and waits the same fixed 10s.
4. Hub side — tower_governor is configured without .use_headers() (crates/towonel-node/src/hub/api/mod.rs), so the 429 carries no Retry-After header, only the body text Wait for Ns. /v1/bootstrap also shares one per-IP bucket with all /v1/auth/* routes (rate_limited_routes). The per-tenant limit is env-configurable (TOWONEL_HUB_RATE_LIMIT_PER_SEC=5, _BURST=30, #44) but N replicas x 10s lockstep still saturates it.

Requested changes

Agent (primary):

1. Make the topology-refresh interval configurable via env var (e.g. TOWONEL_AGENT_TOPOLOGY_REFRESH_SECS), default 10s.
2. Add jitter (e.g. ±15%) so replicas don't poll in lockstep.
3. Honor the 429 wait hint on the bootstrap poll (parse Wait for Ns, or the Retry-After header if the hub starts sending one) and delay the next poll accordingly.

Hub (complementary):
4. Enable tower_governor's .use_headers() so 429s carry a standard Retry-After header (usable by every client).
5. Consider a dedicated bucket for /v1/bootstrap (separate from auth) and/or a per-tenant budget sized for realistic HA replica counts.

Operator-side mitigation already shipped

The operator now honors the hub's wait hint on its own hub calls (jacaudi/towonel-operator#59), so it won't add to the storm — but it can't change the agent's poll cadence, which is where the volume comes from.

Environment

Hub hub.towonel.dev; agent image 1.0.1; 2 replicas, region CA. Refs (main): crates/towonel-agent/src/main.rs (TOPOLOGY_REFRESH_INTERVAL), crates/towonel-agent/src/hub_client.rs (check_response), crates/towonel-node/src/hub/api/mod.rs (governor), crates/towonel-node/src/config.rs (RateLimitConfig). Observed 2026-07-09.