Skip to content

Stop a mock deployment passing as a real one - #25

Merged
16francej merged 2 commits into
mainfrom
doctor-drop-false-serving-claim
Jul 30, 2026
Merged

Stop a mock deployment passing as a real one#25
16francej merged 2 commits into
mainfrom
doctor-drop-false-serving-claim

Conversation

@16francej

@16francej 16francej commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #22.

HARNESS unset means mock, and mock answers every message with canned text while calling no model provider. Nothing said so. Scaffold with --model-provider openrouter, delete HARNESS from env.core, and:

  • qm check passes clean
  • qm doctor passes — and reported OPENROUTER_API_KEY accepted, serving openrouter/auto
  • qm up deploys

The operator gets a stack that signs people in, accepts messages, and replies with nothing real behind it. Same shape as the bug #22 removed, in its quieter form: that one at least refused the first message out loud.

Two changes

Doctor stops claiming a model it never checked. The key probe is real; the model half was a guess, since doctor never looks at the harness. Now:

- base model provider openrouter: OPENROUTER_API_KEY accepted, serving openrouter/auto
+ base model provider openrouter: OPENROUTER_API_KEY accepted

MODEL_PROVIDER_BASE_MODELS existed only to render the deleted half and now has no production caller, so it goes, along with the drift test pinning it to the core registry — a table nothing reads cannot drift into anything. MODEL_PROVIDER_HARNESSES and its drift test stay; validateModelProvider reads that one.

All three gates name a mock deployment. qm check and qm doctor share one mockHarnessWarning so the wording cannot drift, and core warns at boot when NODE_ENV=production resolves to mock, next to the existing sprites warning.

qm check is the one that matters: it never reaches doctorCommon — only up, fly deploy and aws deploy do — so it was the gate every operator runs and the only one that never mentioned this.

! env.core.HARNESS is unset, which means "mock": this deployment answers every message
  with canned text and calls no model provider. Set it to "pi" for a deployment that
  runs real agent turns.

Scope

Warnings only. Mock stays reachable and unblocked: deployment.md offers it for a local test drive, and 101 of the root suite's 369 test files reach it through testConfig's default.

Two things deliberately left out:

  • Refusing unset HARNESS in production. Cleaner than a warning — deploy/core/Dockerfile pins ENV NODE_ENV=production, so it is a precise signal, and only two test fixtures would need a line. It is a behaviour change and wants its own review.
  • PUBLIC_API_URL silently dropping out of the required set under mock (it is gated on pi/opencode/codex). That is a secret-catalog change, not a warning, and moves what qm up demands.

Verification

Warning confirmed firing through the real CLI on the exact scenario above. CLI suite 485/485, affected core tests 53/53, tsc clean on core, CLI and contract projects; eslint, oxlint --deny-warnings, prettier --check, knip clean.

The base-model step reported "OPENROUTER_API_KEY accepted, serving
openrouter/auto". The key check is real, but the model half is a guess:
doctor never looks at the harness, so on a stack where HARNESS is unset or
mock the line names a model no turn will ever reach. That deployment answers
every message with canned output, and the one gate meant to catch it prints
a green line implying otherwise.

Report only what was actually proven — the provider accepted the key.

MODEL_PROVIDER_BASE_MODELS existed solely to render that half of the message
and now has no caller, so it goes with it, along with the drift test that
pinned it to the core registry. A table nothing reads cannot drift into
anything. Telling an operator which model a harness will really serve needs
the harness in hand; that belongs with the wider mock-deployment warning,
not with a constant kept alive by its own test.
HARNESS unset means mock, and mock answers every message with canned text
while calling no model provider. Nothing said so: `qm check` passed clean,
`qm doctor` passed, `qm up` deployed, and the operator got a stack that signs
people in, accepts messages, and replies with nothing real behind it. That is
the failure this branch's parent set out to remove, in its quieter form — the
provider bug at least refused the first message out loud.

All three gates now say it. `qm check` and `qm doctor` share one
mockHarnessWarning so the wording cannot drift, and core warns at boot when
NODE_ENV=production resolves to mock, next to the existing sprites warning.
`qm check` matters most: it never reaches doctorCommon, which only `up`, fly
deploy and aws deploy do, so it was the one gate an operator always runs and
the one that never mentioned this.

Warnings only. Mock stays reachable and unblocked — deployment.md offers it
for a local test drive, and 101 of the root suite's test files reach it
through testConfig's default.
@16francej
16francej force-pushed the doctor-drop-false-serving-claim branch from df8f38d to 0e9b981 Compare July 30, 2026 20:18
@16francej 16francej changed the title Stop qm doctor claiming a model the deployment may never serve Stop a mock deployment passing as a real one Jul 30, 2026
@16francej
16francej merged commit 3300ec4 into main Jul 30, 2026
15 checks passed
@16francej
16francej deleted the doctor-drop-false-serving-claim branch July 30, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant