A turn's outcome isn't recorded faithfully: silent means two opposite things, and suppressed turns are absent from turn_metrics

[ianTPE]

A turn's outcome isn't recorded faithfully: silent means two opposite things, and suppressed turns are absent from turn_metrics entirely

I built a harness task against the Slack ambient path and it produced a wrong
verdict three times running, each time because the durable record does not say
what happened to a turn. The bugs were mine and are fixed, but each one was
downstream of the same thing: an operator, a dashboard, or an alert reading
qm's own tables cannot distinguish "the agent answered" from "the agent was
suppressed."

Numbers below are from my deployment, ~1,200 turns.

1. status: "silent" covers both suppression and success

src/core/orchestrator.ts:1581 — the ambient reply gate declined, nothing
happened:

return reactions
  ? { status: "react", sessionId: session.id, reactions }
  : { status: "silent", sessionId: session.id };

src/core/orchestrator.ts:2927 — the turn completed normally and delivered its
reply through a surface tool, which is how every ordinary Slack answer comes
back:

} else if (input.surfaceTools && surfaceToolDeps && !strictReadOnly) {
  finalResult = { status: "silent", sessionId: session.id, ...(result.stopped ? { stopped: true } : {}) };
}

There is a third producer at :2926 — a scheduled (poll-fired) turn where the
model itself chose to say nothing:

} else if (isPollFire && !outbound.attachments.length && isSilentPollReply(reply)) {
  finalResult = { status: "silent", sessionId: session.id };
}

That one is at least close to genuine silence, but it lands in the record as
the same value, so it cannot be separated from suppression either.

Same value, opposite outcomes. On this deployment:

runs.result status   ok 496 | silent 106 | refused 15 | failed 14 | react 3

Those 106 are unresolvable from the record. Some are agents that answered a
colleague; some are messages qm decided not to answer at all. Nothing stored
separates them.

I hit this concretely: a control run where qm replied correctly in-thread came
back silent, and scoring on that field marked a working exchange as a failure.
Reading the Slack transcript instead has the mirror problem — it cannot tell a
declined turn from a model that ran and chose to say nothing. Neither source
answers the question on its own.

2. A declined turn never reaches the metrics sink

The gate returns at :1565. The metrics emit is at :2736, ~1,200 lines later
in the same function. So a turn the gate declines writes:

• no turn_metrics row
• no session_llm_requests row (the detection call is passed only
  recordModelCall, which lands in the in-memory ring in
  src/model/model-gateway.ts and is never persisted)

Its sole durable trace is runs.result.

turn_metrics status   capture 600 | ok 600

No react, no silent, no refused, no failed — in 1,200 rows. Which brings
me to:

3. turn_metrics.status is hardcoded, so even the recorded turns don't carry their outcome

// src/core/orchestrator.ts:2740
status: pausing ? "paused" : "ok",

finalResult is computed a hundred lines below and never reaches the sink. The
arithmetic confirms it: 496 ok + 106 silent in runs ≈ the 600 ok in
turn_metrics. Turns that returned silent are recorded as ok.

So the natural table for an operational dashboard — the one with total_ms,
ttft_ms, model_calls and the rest — reports every turn it knows about as
successful, and doesn't know about the suppressed ones at all.

4. detect_ms exists but can only ever record a "yes"

turn_metrics.detect_ms populated:  1 / 1200

The column is declared in src/admin/postgres-metrics-sink.ts:31 and assigned
at :2736. Because the declining path returns at :1565, detect_ms is
written only when the gate said respond — the case least in need of
investigation. Every declined decision, and its latency, is dropped.

Why this is worth fixing beyond tidiness

The ambient gate is a model call that decides whether a person gets an answer.
On this deployment it silently dropped an answer to a question qm itself had
asked and promised to report on. There is currently no query that would surface
that class of event, no metric that would trend it, and nothing that records
which model made the call — judgeModelId resolves to
auxiliaryModelFor(orgBaseModelId()), so for an Anthropic org model the decision
runs on claude-haiku-4-5, but that is derived from source, not observed.

An operator who suspects qm is ignoring people has, today, only runs.result
and a status that means two opposite things.

Suggestions

1. Split the two silent cases. delivered (or reuse ok) for a surface-tool
   delivery, and keep silent for suppression — or add a reason alongside.
   One value for both makes every downstream consumer wrong by default.
2. Emit a turn_metrics row on the early returns, so declined turns exist in
   the same table as the rest, with detect_ms and the gate's verdict.
3. Carry finalResult.status into the sink rather than hardcoding "ok".
4. Persist the detection call's model on session_llm_requests like any other
   model call, so which model made the decision is answerable after the fact.

(1) and (3) look small and would do most of the work.

Happy to test a branch; I have a live deployment and a harness that drives this
path — ambient-reply.mjs, which
also documents the three ways this record misled me while I was building it.
Related: #607 (the gate's decision itself) and #608 (the same hand-off seen from
another session).