Skip to content

feat(observability): correlate agent relay delivery - #4769

Open
MajorTal wants to merge 1 commit into
block:mainfrom
MajorTal:codex/agent-response-diagnostics
Open

feat(observability): correlate agent relay delivery#4769
MajorTal wants to merge 1 commit into
block:mainfrom
MajorTal:codex/agent-response-diagnostics

Conversation

@MajorTal

@MajorTal MajorTal commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Trying to debug why my agents sometimes do not respond, I realised the current diagnostics are insufficient. Which is why this PR adds a focused correlation trail across the relay, Redis pub/sub, and the ACP harness.

Today a delayed response can look like a generic WebSocket stream ending followed by a later reconnect. The logs do not preserve the Close code/reason, join the client socket to the server-side connection, or show whether Redis and local WebSocket fan-out handled the event live.

What changed

  • Return an opaque x-buzz-connection-id on the WebSocket upgrade so ACP and relay logs identify the same socket.
  • Record ACP connection generations, Close code/reason, error/EOF/ping-timeout exits, and event receipt age/replay hints.
  • Record Redis publish subscriber counts and relay fan-out recipient/drop counts using the existing event ID.
  • Add Redis subscriber connection, disconnect, reconnect, and topic gauges, and enable buzz_pubsub=info in the relay's default log/OTEL filters.

The new diagnostics contain IDs, timestamps, and counts only—no message contents, auth material, or private keys. Delivery behavior is otherwise unchanged.

What this should answer next time

  • Did the ACP socket close cleanly, time out, reset, or disappear without a Close frame?
  • Which relay-side connection handled that exact socket?
  • Did Redis report any subscribers when the event was published?
  • Did the receiving relay match any local WebSocket subscriptions?
  • Did ACP receive the event live or only after reconnect/replay?

Validation

  • cargo fmt --all --check
  • cargo clippy -p buzz-acp -p buzz-pubsub -p buzz-relay --all-targets -- -D warnings
  • cargo test -p buzz-acp --lib — 669 passed
  • cargo test -p buzz-pubsub --lib — 24 passed, 11 ignored
  • Targeted relay and diagnostics tests pass
  • cargo test -p buzz-relay --lib — 839 passed, 37 ignored; demo_join_forwarded_arm_round_trips_echo fails with the same 504 vs 200 assertion on this branch and an untouched detached origin/main worktree

@MajorTal
MajorTal force-pushed the codex/agent-response-diagnostics branch from c25ef5f to a7f0cc2 Compare August 4, 2026 19:03
@MajorTal
MajorTal marked this pull request as ready for review August 7, 2026 12:41
@MajorTal
MajorTal requested a review from a team as a code owner August 7, 2026 12:41
Signed-off-by: Tal Weiss <major.tal@gmail.com>
@MajorTal

Copy link
Copy Markdown
Contributor Author

Filed #5472 for the diagnostic gap this addresses, per CONTRIBUTING.md's issue-first guidance. The open judgment call is log volume — making this useful by default means enabling buzz_pubsub=info in the relay's default filters. That trade is better decided there than in a diff review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant