The ambient reply gate discards direct answers to questions qm itself asked, and fails closed

[ianTPE]

The ambient reply gate discards direct answers to questions qm itself asked, and fails closed

qm was asked to chase a colleague for an answer and report back. The colleague
answered, in the thread, under qm's own question. qm reacted with 👍 and never
reported. The requester was told an hour later that no answer had arrived.

This is not the model that serves the scope making a judgement call — it is a
separate gate, on a separate model, that runs only on messages qm was not
explicitly @-mentioned in, and that defaults to staying quiet whenever it is
uncertain or broken.

(Names below are from a test scenario on my own deployment, translated. The
timestamps, ids and flags are verbatim from Postgres.)

What happened

A private channel. founder asks qm to chase admin and report back within the
hour. Times are Asia/Taipei.

time	who	message	mentions_self	handled
22:13	founder	"@qm John starts tomorrow, ask @Admin whether his insurance enrolment is done … report back to me in an hour"	t	t
22:13	qm	"@Admin John starts tomorrow, has his enrolment gone through? … @founder I'll report back in an hour"	—	—
23:15	qm	"@founder an hour's up: no reply from @Admin yet"	—	—
23:26	admin	"Sorry, didn't see the notification. Filed Friday, effective 8/17, all good."	f	t
23:50	admin	"<@B0BND0YAX4J> did admin answer about the insurance?"	f	f
00:13	admin	"Didn't I reply? Why are you saying I didn't?"	f	t

The 23:26 row is the one that matters. handled = t — qm ingested it and ran a
turn. That turn's stored result:

{"status":"react","sessionId":"a11e5cb2-…","reactions":["+1"]}

It read the answer to its own question, gave it a thumbs-up, and dropped the
commitment it had made 73 minutes earlier.

Why the gate saw it at all

src/core/orchestrator.ts:398 marks a turn ambient when it arrives without an
explicit mention, and :1540 routes ambient turns through an extra model call
before any real work happens:

if (ambientTurn && deps.harness.models.shouldRespond && …) {
  const decision = await deps.harness.models.shouldRespond({ … });
  …
  if (!decision.respond) {            // :1565
    return reactions ? { status: "react", … } : { status: "silent", … };
  }

A message that @-mentions qm never reaches this branch. So the reliability of a
tracked hand-off depends on whether the person answering happens to type a
mention — which is precisely the thing qm told them they didn't need to do
("just reply in this thread").

The prompt already specifies the right answer

TURN_DETECTION_PROMPT_HEAD (src/harness/pi-harness.ts:131) covers this case
twice, explicitly:

:140   "it answers a question you just asked, says yes/no/go ahead/that/this in response
        to your prior message"                                              → YES
:159   "is posted in a thread the assistant itself STARTED … Someone replying under your
        own message is almost always talking to you, even without naming you"→ YES

and ships a near-identical worked example:

"yes, send it" after you offered to send something → YES (a direct answer to you, no mention needed)

Admin's message was a direct answer to a question qm posted in that thread. Two
rules and one example all point at YES. The classifier returned REACT.

So this is not a gap in the prompt to be patched with another bullet. A
deterministic condition — qm asked a question in this thread and has not
recorded an answer — is being delegated to a probabilistic call that can, and
here did, get it wrong.

Two ways the gate fails closed

Uncertainty resolves to silence. pi-harness.ts:188: "When genuinely
unsure, prefer NO — a good colleague would rather stay quiet than barge in."
Sound default for chit-chat in a busy channel. Wrong default when qm is holding
an unfulfilled promise in the same thread, because there the cost of staying
quiet is not a missed pleasantry, it is a broken commitment that nobody is told
about.

A failed detection call also resolves to silence. claude-harness.ts:895:

} catch (error) {
  swallow("claude: detect", error);
  return { respond: false };
}

Any provider hiccup on the detect call — the same class of failure as #602 — silently converts into "don't answer."
Nothing is posted, nothing is surfaced to an operator, and the turn records a
silent status.

Worth noting the gate also runs on judgeModelId, not the model the scope is
pinned to. Whatever model an admin evaluated and selected for a channel is not
the model deciding whether that channel's messages deserve an answer.

Suggested fix

Skip the ambient gate — or force respond: true — when qm has an outstanding
commitment in the thread: it asked a question, promised a report, or set a
deadline that has not been discharged. That state is already known; the thread
session here is literally titled "Chase … status, report back", and the
scheduled report at 23:15 fired from it.

Failing that, at minimum:

1. catch → { respond: false } should not be indistinguishable from a
   considered "no". Treat a detection failure as "respond" (the safe direction
   when someone is waiting on you), or surface it.
2. The "prefer NO when unsure" tie-break should invert in threads qm started or
   asked a question in — the prompt already has the rule, it just loses to the
   tie-break.

A related consequence is filed as #608: once the update was not pushed, the
requester could not pull it either.

I have the full transcript, run rows and session ids and can supply them, or
test a branch against a live deployment.

[yzxcj797]

Confirmed the fail-closed reading of the code paths you cite — adding two observations from the source that sharpen where a fix can land cheapest, since the full commitment-aware gate is a design decision:

1. The catch → respond: false sites are several, and they all collapse into silent. Beyond claude-harness.ts:895, the pi detect path and the orchestrator's ambient gate wrapper have the same shape (detection errors swallowed, turn completes silent). So a provider blip doesn't just lose one message — during a degraded window every ambient message in every scope is silently dropped, which compounds with #602: the model failures that cause detect to fail are exactly when you'd want the ambient gate to stop filtering. Any fix that distinguishes "detect said no" from "detect didn't answer" has to change the return type of the detect seam (a tri-state), not just the catch blocks — otherwise the orchestrator can't tell them apart at orchestrator.ts:1540.

2. The judgeModelId observation generalizes into a config question the maintainers should decide once. The gate also runs for scopes whose admin pinned a specific model for the work; the judge runs regardless. If the direction taken is "gate less often" (commitment-aware skip), the judge-model question mostly evaporates; if it's "gate better", the judge should at least be per-scope configurable. Worth bundling that decision with whichever fix lands.

On the suggested-fix ladder, my read of the blast radius:

• Commitment-aware skip — the principled fix, and the state genuinely is already known (the scheduled 23:15 report fired from the same thread session). But it needs a definition of "outstanding commitment" the orchestrator can evaluate at gate time (pending scheduled report for this thread? last assistant message contains an unanswered question — parseable how?), and each candidate definition has false positives (a question qm asked that WAS answered in a thread the classifier missed). Not a one-liner; worth doing right.
• Fail-open on detect error — one-line-per-site but wrong as a blanket: during a provider outage every ambient message in every channel would suddenly get answers, which is its own incident. The tri-state from (1) is the prerequisite, then the orchestrator can fail open per-scope (or only when a scheduled task references the thread).
• Tie-break inversion in own threads — cheapest and directly targets this transcript (both prompt rules already say YES; the tie-break is what kills it). A threadStartedBySelf flag into the detect prompt is mechanical and low-risk.

If the maintainers pick a direction I'm happy to take the implementation — (3) plus the tri-state from (1) is a coherent, small first slice.

[ianTPE]

Following up on the cost objection from discussion — "doesn't tag-free in-thread listening just burn tokens?" — because it sharpens the fix rather than opposing it.

Accuracy note on current behavior, because it reframes the objection. The Slack path already doesn't gate every untagged message: an unmentioned channel message is dispatched only when it's a thread reply in a thread the bot already has a stake in, and the shouldRespond gate runs on that narrower set. (DMs dispatch unconditionally and skip the gate.) So it's already partly decoupled from channel chatter. Commitment-scoping tightens "bot-stake threads" to "threads with an open commitment" — a narrowing of existing scoping, not a jump from "all channel noise."

Cost, precisely:

• Whether a thread is in scope is a lookup — (installation, channel, thread_ts) → zero-or-more open commitment IDs — so once durable commitment state exists, the routing decision spends no model tokens (still an event-ingestion + DB read).
• Whether a message is the answer still needs a model, on each message received while its thread has ≥1 open commitment (all commitments in a thread evaluated together, not once per commitment). That cost scales with per-message context size, not channel volume — decoupled from total chatter, but not zero.
• Requiring @qm on every reply is genuinely the cheapest, but unenforceable: reliability then hinges on the responder remembering to tag — the exact thing that failed here (admin didn't, then asked "didn't I reply?").

So cost favors scoping, but reliability is the real case: the gate fails closed by construction, so any unsure or errored evaluation drops the answer. Cheaper is a bonus, not the headline.

Two honest caveats on the scoped design:

• It relocates fail-closed, it doesn't remove it. Correctness now depends on the commitment being durably created and bound — and Slack and the commitment DB can't share a transaction, while a bot-opened root thread_ts is only known after Slack accepts the post. So this needs an idempotent create → post → bind → open saga with an outbox and repair, not an "atomic write." A more observable failure surface than a per-message classifier, but not a free one.
• Today's scheduled report is evidence a commitment existed, not a structured commitment record. The state has to be built; it isn't already sitting there typed.

The gap scoping still has, and the backstop. Push (in-thread, scoped to open commitments) is blind to answers that don't land in the canonical thread — a top-level reply, a new thread, a DM. Pair it with pull reconciliation at the deadline: before reporting "no answer", re-read the canonical thread — with pagination; current retrieval caps at 200 messages with no cursor loop, which a busy thread overflows — and re-evaluate. This is a durable retrying backstop, not a guarantee: a reread recovers a dropped event or a missed answer-write only if the commitment/job persisted and Slack history is reachable, and it does not recover cross-thread answers (out of scope unless a bounded search over the expected responder's messages since asked_at is added).

The issue's own timeline has a tail worth designing for: the answer arrived after the deadline. A deadline-only pull correctly reports "no answer yet", but if the later push also fails the answer is lost — so after reporting overdue, the commitment must stay open through a grace window for late-answer correction.

Net architecture (TL;DR): bypass shouldRespond only for messages that map to open commitments, evaluated together; reconcile at the deadline with pagination and keep the commitment open through a grace window; idempotency/outbox throughout. Bypassing only for commitment-mapped messages matters — a blanket bypass for every qm-started thread would let unrelated chatter trigger full agent turns (and tools).

This is a state machine, not a prompt bullet. Mechanics that must be specified: evaluator outcomes kept distinct (answered / not_answer / clarification / error — never collapse a provider error into not_answer, cf. #602); answer-vs-deadline races serialized with CAS/versioning; expected-respondent identity matched so an unrelated participant doesn't auto-resolve the commitment; partial answers and multiple commitments per thread handled; edits/deletes treated as state-changing (a deleted or materially edited accepted answer may reopen the commitment); and a commitment lifecycle so the tracked-thread set stays bounded. All of it operator-observable: reconciliation failures, overdue commitments, evaluator errors, undelivered reports.