Stop a mock deployment passing as a real one - #25
Merged
Conversation
The base-model step reported "OPENROUTER_API_KEY accepted, serving openrouter/auto". The key check is real, but the model half is a guess: doctor never looks at the harness, so on a stack where HARNESS is unset or mock the line names a model no turn will ever reach. That deployment answers every message with canned output, and the one gate meant to catch it prints a green line implying otherwise. Report only what was actually proven — the provider accepted the key. MODEL_PROVIDER_BASE_MODELS existed solely to render that half of the message and now has no caller, so it goes with it, along with the drift test that pinned it to the core registry. A table nothing reads cannot drift into anything. Telling an operator which model a harness will really serve needs the harness in hand; that belongs with the wider mock-deployment warning, not with a constant kept alive by its own test.
HARNESS unset means mock, and mock answers every message with canned text while calling no model provider. Nothing said so: `qm check` passed clean, `qm doctor` passed, `qm up` deployed, and the operator got a stack that signs people in, accepts messages, and replies with nothing real behind it. That is the failure this branch's parent set out to remove, in its quieter form — the provider bug at least refused the first message out loud. All three gates now say it. `qm check` and `qm doctor` share one mockHarnessWarning so the wording cannot drift, and core warns at boot when NODE_ENV=production resolves to mock, next to the existing sprites warning. `qm check` matters most: it never reaches doctorCommon, which only `up`, fly deploy and aws deploy do, so it was the one gate an operator always runs and the one that never mentioned this. Warnings only. Mock stays reachable and unblocked — deployment.md offers it for a local test drive, and 101 of the root suite's test files reach it through testConfig's default.
16francej
force-pushed
the
doctor-drop-false-serving-claim
branch
from
July 30, 2026 20:18
df8f38d to
0e9b981
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #22.
HARNESSunset meansmock, and mock answers every message with canned text while calling no model provider. Nothing said so. Scaffold with--model-provider openrouter, deleteHARNESSfromenv.core, and:qm checkpasses cleanqm doctorpasses — and reportedOPENROUTER_API_KEY accepted, serving openrouter/autoqm updeploysThe operator gets a stack that signs people in, accepts messages, and replies with nothing real behind it. Same shape as the bug #22 removed, in its quieter form: that one at least refused the first message out loud.
Two changes
Doctor stops claiming a model it never checked. The key probe is real; the model half was a guess, since doctor never looks at the harness. Now:
MODEL_PROVIDER_BASE_MODELSexisted only to render the deleted half and now has no production caller, so it goes, along with the drift test pinning it to the core registry — a table nothing reads cannot drift into anything.MODEL_PROVIDER_HARNESSESand its drift test stay;validateModelProviderreads that one.All three gates name a mock deployment.
qm checkandqm doctorshare onemockHarnessWarningso the wording cannot drift, and core warns at boot whenNODE_ENV=productionresolves to mock, next to the existing sprites warning.qm checkis the one that matters: it never reachesdoctorCommon— onlyup,fly deployandaws deploydo — so it was the gate every operator runs and the only one that never mentioned this.Scope
Warnings only. Mock stays reachable and unblocked:
deployment.mdoffers it for a local test drive, and 101 of the root suite's 369 test files reach it throughtestConfig's default.Two things deliberately left out:
HARNESSin production. Cleaner than a warning —deploy/core/DockerfilepinsENV NODE_ENV=production, so it is a precise signal, and only two test fixtures would need a line. It is a behaviour change and wants its own review.PUBLIC_API_URLsilently dropping out of the required set under mock (it is gated on pi/opencode/codex). That is a secret-catalog change, not a warning, and moves whatqm updemands.Verification
Warning confirmed firing through the real CLI on the exact scenario above. CLI suite 485/485, affected core tests 53/53,
tscclean on core, CLI and contract projects;eslint,oxlint --deny-warnings,prettier --check,knipclean.