Summary
When an agent does not respond, the logs do not contain enough information to tell where the event stopped. The observable symptom is a WebSocket stream ending, then a reconnect some time later — which looks identical whether the socket was closed cleanly, timed out, was reset, or the event was never fanned out to it in the first place.
What's missing
Chasing one of these today, the following questions have no answer in the logs:
- Did the ACP socket close cleanly, time out, reset, or vanish without a Close frame? The Close code and reason are not preserved, so every ending looks the same.
- Which relay-side connection handled that exact socket? There is no identifier shared between the client's view of the socket and the server's, so the two sides' logs cannot be joined.
- Did Redis report any subscribers when the event was published? Publishing to zero subscribers and publishing to a subscriber that then dropped the event are indistinguishable.
- Did the receiving relay match any local WebSocket subscriptions? Fan-out recipient and drop counts are not recorded.
- Did ACP receive the event live, or only later via reconnect/replay? Without receipt age there is no way to tell a slow path from a replayed one.
Each of these is individually small; together they mean the difference between "the agent is broken" and "the event never reached the socket" is not derivable from the logs, and every occurrence turns into a live-reproduction exercise.
Proposed shape
A correlation trail across the three hops (relay → Redis pub/sub → ACP harness), using IDs and counts only:
- return an opaque connection id on the WebSocket upgrade so ACP-side and relay-side logs can be joined to the same socket
- record ACP connection generations, Close code/reason, and the distinct exit paths (error, EOF, ping timeout) rather than collapsing them
- record event receipt age and a replay hint, so live delivery is distinguishable from replay
- record Redis publish subscriber counts, and relay fan-out recipient/drop counts, keyed on the existing event id
- add Redis subscriber connect/disconnect/reconnect and topic gauges
Two constraints I'd hold this to: it should carry no message contents, auth material, or keys — ids, timestamps, and counts only — and it should not change delivery behavior, only describe it.
The one judgment call worth a maintainer's opinion is log volume: making this useful by default means enabling buzz_pubsub=info in the relay's default filters, which is a deliberate increase in baseline output. Diagnostics that are off by default tend not to be on when the incident happens, but that trade is the maintainers' call, not mine.
Status
Filed after the fact: #4769 implements the above and is open. Filing here per CONTRIBUTING.md's issue-first guidance so the approach — particularly the default-log-level question and whether the connection id should be exposed on the upgrade response — can be settled separately from reviewing the diff.
Summary
When an agent does not respond, the logs do not contain enough information to tell where the event stopped. The observable symptom is a WebSocket stream ending, then a reconnect some time later — which looks identical whether the socket was closed cleanly, timed out, was reset, or the event was never fanned out to it in the first place.
What's missing
Chasing one of these today, the following questions have no answer in the logs:
Each of these is individually small; together they mean the difference between "the agent is broken" and "the event never reached the socket" is not derivable from the logs, and every occurrence turns into a live-reproduction exercise.
Proposed shape
A correlation trail across the three hops (relay → Redis pub/sub → ACP harness), using IDs and counts only:
Two constraints I'd hold this to: it should carry no message contents, auth material, or keys — ids, timestamps, and counts only — and it should not change delivery behavior, only describe it.
The one judgment call worth a maintainer's opinion is log volume: making this useful by default means enabling
buzz_pubsub=infoin the relay's default filters, which is a deliberate increase in baseline output. Diagnostics that are off by default tend not to be on when the incident happens, but that trade is the maintainers' call, not mine.Status
Filed after the fact: #4769 implements the above and is open. Filing here per CONTRIBUTING.md's issue-first guidance so the approach — particularly the default-log-level question and whether the connection id should be exposed on the upgrade response — can be settled separately from reviewing the diff.