fix(relay): exempt ephemeral events from Messages quota; add limit_type observability - #4902
Closed
wpfleger96 wants to merge 1 commit into
Closed
fix(relay): exempt ephemeral events from Messages quota; add limit_type observability#4902wpfleger96 wants to merge 1 commit into
wpfleger96 wants to merge 1 commit into
Conversation
…pe observability
Ephemeral events (kinds 20000–29999) are never persisted by storage, yet
WS admission billed them against the per-minute durable Messages budget.
With buzz-acp publishing up to 90 observer frames/min + 20 typing
indicators/min/channel + 1 presence/min, agents consumed ~111 of their
120/min Messages budget on pure telemetry, causing repeated 40s quota
stalls that blocked real message delivery.
Changes:
- WS admission now skips LimitType::Messages for ephemeral kinds, using
the existing is_ephemeral() range predicate (same one storage uses to
refuse persistence — admission and storage now agree by construction).
Ephemeral events still count against WsEvents so per-second burst
protection remains intact.
- Add agent_ws_events_per_sec to RateLimitConfig (env:
BUZZ_RATE_LIMIT_AGENT_WS_EVENTS_PER_SEC). Agents previously inherited
human_ws_events_per_sec silently. Default matches human default (10/s)
so this is behavior-neutral at merge; tune on builderlab once
limit_type instrumentation data is available.
- Delete three dead tier fields that were defined and env-loadable but
enforced nowhere: agent_elevated_messages_per_min,
agent_platform_messages_per_min, agent_standard_api_calls_per_min.
- Add limit_type to NOTICE/CLOSED rejection text (format: 'quota exceeded
({limit_type}); retry in {N}s' — the 'retry in Ns' phrase is preserved
for client parsers) and to buzz_admission_rejections_total metric as a
new label on both WS and HTTP paths.
Post-deploy validation: ACP Messages rejections should drop to ~0;
any residual >5s retry hint on the WS path indicates an unenumerated
durable WS publisher.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Relay WS admission bills every EVENT kind against the per-pubkey durable-message quota (), regardless of whether the event is persisted. With buzz-acp publishing up to 90 observer frames/min + 20 typing indicators/min/channel + 1 presence/min, agents consumed ~111 of their 120/min
Messagesbudget on pure telemetry — leaving only 9 msg/min for real messages and causing repeated 40s quota stalls.Additionally, agents silently inherited the human WS burst budget with no dedicated config field, and three tier-config fields (
agent_elevated_messages_per_min,agent_platform_messages_per_min,agent_standard_api_calls_per_min) were defined, env-loadable, and enforced nowhere.Changes
Core fix —
crates/buzz-relay/src/connection.rsEphemeral events (kinds 20000–29999) now skip
LimitType::Messagesin WS admission. Uses the existingis_ephemeral()range predicate frombuzz-core— the same onebuzz-dbuses to refuse persistence, so admission and storage agree by construction. Ephemeral events still count againstWsEvents(per-second burst protection unchanged).Observability —
crates/buzz-relay/src/connection.rs,src/api/bridge.rslimit_typeis now included in:rate-limited: quota exceeded (ws_events); retry in 5srate-limited: quota exceeded (api_calls); retry in 3sbuzz_admission_rejections_totalmetric labelThe
retry in {N}sphrase is preserved intact — ACP and CLI both parse it.Agent WS budget —
crates/buzz-auth/src/rate_limit.rs,crates/buzz-relay/src/config.rsAdded
agent_ws_events_per_sectoRateLimitConfigwith env overrideBUZZ_RATE_LIMIT_AGENT_WS_EVENTS_PER_SEC. Default matches the human default (10/s) so this is behavior-neutral at merge. Tune on builderlab oncelimit_typeinstrumentation data establishes the right operating value.Dead config cleanup —
crates/buzz-auth/src/rate_limit.rs,crates/buzz-relay/src/config.rsDeleted three fields enforced nowhere:
agent_elevated_messages_per_min,agent_platform_messages_per_min,agent_standard_api_calls_per_min. Removal is grep-clean — no dangling readers in the owned crates.Tests
New unit tests cover:
LimitType::as_str()values are stable (breaking change if they change — they appear in metric labels)RateLimitConfig::default()hasagent_ws_events_per_secequal to human defaultsend_admission_result: NOTICE text names the limit type and preservesretry in Nsphrase for bothMessagesandWsEvents;Ok(())sends nothing; sub-scoped rejection emits CLOSEDBUZZ_RATE_LIMIT_AGENT_WS_EVENTS_PER_SECenv override works and rejects zeroFull suite: 844 passing / 1 pre-existing failure (
mesh_demo— reproduces onorigin/mainbefore this branch).Post-deploy validation
After deploy,
buzz_admission_rejections_total{reason="quota",limit_type="messages"}for agent pubkeys should drop to ~0. Any residual>5sretry hint on the WS path indicates an unenumerated durable WS publisher.