Skip to content

fix(relay): exempt ephemeral events from Messages quota; add limit_type observability - #4902

Closed
wpfleger96 wants to merge 1 commit into
mainfrom
duncan/ephemeral-admission-fix
Closed

fix(relay): exempt ephemeral events from Messages quota; add limit_type observability#4902
wpfleger96 wants to merge 1 commit into
mainfrom
duncan/ephemeral-admission-fix

Conversation

@wpfleger96

Copy link
Copy Markdown
Member

Problem

Relay WS admission bills every EVENT kind against the per-pubkey durable-message quota (), regardless of whether the event is persisted. With buzz-acp publishing up to 90 observer frames/min + 20 typing indicators/min/channel + 1 presence/min, agents consumed ~111 of their 120/min Messages budget on pure telemetry — leaving only 9 msg/min for real messages and causing repeated 40s quota stalls.

Additionally, agents silently inherited the human WS burst budget with no dedicated config field, and three tier-config fields (agent_elevated_messages_per_min, agent_platform_messages_per_min, agent_standard_api_calls_per_min) were defined, env-loadable, and enforced nowhere.

Changes

Core fix — crates/buzz-relay/src/connection.rs

Ephemeral events (kinds 20000–29999) now skip LimitType::Messages in WS admission. Uses the existing is_ephemeral() range predicate from buzz-core — the same one buzz-db uses to refuse persistence, so admission and storage agree by construction. Ephemeral events still count against WsEvents (per-second burst protection unchanged).

Observability — crates/buzz-relay/src/connection.rs, src/api/bridge.rs

limit_type is now included in:

  • WS rejection NOTICE/CLOSED text: rate-limited: quota exceeded (ws_events); retry in 5s
  • HTTP 429 response body: rate-limited: quota exceeded (api_calls); retry in 3s
  • buzz_admission_rejections_total metric label

The retry in {N}s phrase is preserved intact — ACP and CLI both parse it.

Agent WS budget — crates/buzz-auth/src/rate_limit.rs, crates/buzz-relay/src/config.rs

Added agent_ws_events_per_sec to RateLimitConfig with env override BUZZ_RATE_LIMIT_AGENT_WS_EVENTS_PER_SEC. Default matches the human default (10/s) so this is behavior-neutral at merge. Tune on builderlab once limit_type instrumentation data establishes the right operating value.

Dead config cleanup — crates/buzz-auth/src/rate_limit.rs, crates/buzz-relay/src/config.rs

Deleted three fields enforced nowhere: agent_elevated_messages_per_min, agent_platform_messages_per_min, agent_standard_api_calls_per_min. Removal is grep-clean — no dangling readers in the owned crates.

Tests

New unit tests cover:

  • LimitType::as_str() values are stable (breaking change if they change — they appear in metric labels)
  • RateLimitConfig::default() has agent_ws_events_per_sec equal to human default
  • send_admission_result: NOTICE text names the limit type and preserves retry in Ns phrase for both Messages and WsEvents; Ok(()) sends nothing; sub-scoped rejection emits CLOSED
  • BUZZ_RATE_LIMIT_AGENT_WS_EVENTS_PER_SEC env override works and rejects zero

Full suite: 844 passing / 1 pre-existing failure (mesh_demo — reproduces on origin/main before this branch).

Post-deploy validation

After deploy, buzz_admission_rejections_total{reason="quota",limit_type="messages"} for agent pubkeys should drop to ~0. Any residual >5s retry hint on the WS path indicates an unenumerated durable WS publisher.

…pe observability

Ephemeral events (kinds 20000–29999) are never persisted by storage, yet
WS admission billed them against the per-minute durable Messages budget.
With buzz-acp publishing up to 90 observer frames/min + 20 typing
indicators/min/channel + 1 presence/min, agents consumed ~111 of their
120/min Messages budget on pure telemetry, causing repeated 40s quota
stalls that blocked real message delivery.

Changes:
- WS admission now skips LimitType::Messages for ephemeral kinds, using
  the existing is_ephemeral() range predicate (same one storage uses to
  refuse persistence — admission and storage now agree by construction).
  Ephemeral events still count against WsEvents so per-second burst
  protection remains intact.
- Add agent_ws_events_per_sec to RateLimitConfig (env:
  BUZZ_RATE_LIMIT_AGENT_WS_EVENTS_PER_SEC). Agents previously inherited
  human_ws_events_per_sec silently. Default matches human default (10/s)
  so this is behavior-neutral at merge; tune on builderlab once
  limit_type instrumentation data is available.
- Delete three dead tier fields that were defined and env-loadable but
  enforced nowhere: agent_elevated_messages_per_min,
  agent_platform_messages_per_min, agent_standard_api_calls_per_min.
- Add limit_type to NOTICE/CLOSED rejection text (format: 'quota exceeded
  ({limit_type}); retry in {N}s' — the 'retry in Ns' phrase is preserved
  for client parsers) and to buzz_admission_rejections_total metric as a
  new label on both WS and HTTP paths.

Post-deploy validation: ACP Messages rejections should drop to ~0;
any residual >5s retry hint on the WS path indicates an unenumerated
durable WS publisher.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 requested a review from a team as a code owner August 5, 2026 15:29
@wpfleger96

Copy link
Copy Markdown
Member Author

🤖 Superseded by #4912, which carries the same ephemeral-exemption + limit_type change as part of the consolidated single-PR approach. Branch kept — the agent_ws_events_per_sec config and dead tier-field removal from this PR aren't in #4912 and will follow up separately after it lands.

@wpfleger96 wpfleger96 closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant