Skip to content

Document playground deployments in the deployment workflow - #39

Merged
16francej merged 6 commits into
mainfrom
playground-deploy-docs
Jul 31, 2026
Merged

Document playground deployments in the deployment workflow#39
16francej merged 6 commits into
mainfrom
playground-deploy-docs

Conversation

@16francej

@16francej 16francej commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #38, which added playground mode but documented it only in the portal README — an operator following the deploy path could never discover it.

Adds a "Playground mode" subsection to step 3 of the deployment workflow (cli/templates/deployment/deployment.md, materialized into every org's deployment directory by qm init), covering:

  • the env.portal.PORTAL_PLAYGROUND stack snippet
  • the its-own-deployment rule: visitors are ordinary internal principals of the org, so nothing sensitive at org scope, no real connector credentials, no company data
  • the real brakes to set in the same pass: ORG_BUDGET_USD_PER_WINDOW as the hard ceiling, per-principal BUDGET_USD_PER_WINDOW / RATE_LIMIT_PER_WINDOW, and a pinned model via the Admin page's base-model / web-UI model resources
  • the boot refusals and anon restrictions the portal enforces, the scope-GC gap, and a pointer to the portal README for the mechanism

Docs only; CLI init tests pass (17/17).


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Why the CLI version bump

cli/templates/deployment/deployment.md ships inside the @yc-software/qm npm package (it's what qm init materializes into a deployment repo). CI's "CLI version bump" check refuses any change to packaged files without a version bump past the published 0.1.2, so this PR bumps cli/package.json to 0.1.3. Doc template change only — no CLI code changes ride along.

Also: enforce the web-UI model picker at every core entry

Reviewing the docs claim surfaced a bypass: webui-models only gated the explicit per-turn model: option. PUT /v1/runtime-config accepted any model an approved harness supports, so a user could pin an excluded model as a scope override and turns ran it without re-validation.

Fixed at both entries: the route now refuses a model outside the configured picker (400 model_not_enabled), and web turn resolution refuses a resolved scope override outside it (covers overrides that predate a picker change). The org default stays exempt — the picker already treats it as the default selection. Unconfigured picker keeps current behavior.

Verified live against a running dev instance: stale override → 403 refused; PUT excluded model → 400; explicit per-turn excluded model → 403; org default turn → 202 queued. Plus unit/integration tests (turn-options, admin-resources).

The portal README documents the mechanism, but the deployment workflow
never mentioned playground mode, so an operator following the deploy
path could not discover it or the companion settings a public try-it
instance needs: its own dedicated deployment, an org-wide spend
ceiling, per-principal brakes, and a pinned model.
The webui-models picker only gated the explicit per-turn model option.
PUT /v1/runtime-config accepted any model an approved harness supports,
so a scope override could pin a model the admin had excluded, and turns
then ran it without re-validation. The route now refuses a model
outside the configured picker, and web turn resolution refuses a scope
override outside it as well, so overrides that predate a picker change
fail closed instead of running. The org default stays exempt: the
picker semantics already treat it as the default selection.
…ides

The web UI always sends an explicit model — the default selection is
the org base model — so a picker that excluded the org default refused
every default turn from the real client. The enabled set is now the
configured picker plus the org default, the runtime-config route
accepts the org default as an override, and the keep-branch legacy
migration stops minting selections the picker excludes. Adds an
app-level test pinning the stale-override refusal and both exemption
paths, which the unit test alone could not: it survived swapping the
wiring's arguments.
@16francej

Copy link
Copy Markdown
Contributor Author

Fresh-context adversarial review of the picker enforcement ran; disposition:

Fixed

  • Org-default exemption was unreachable for the real web UI (it always sends an explicit model: — the default selection is the org base model — so a picker excluding the org default refused every default turn): the enabled set is now the configured picker ∪ org default at all three entries (per-turn option, stored-override resolution, PUT /v1/runtime-config).
  • keep: true legacy migration could mint an unchecked selection: it now skips migrating a legacy base model the picker excludes.
  • Test gap: added an app-level test driving app.turn end-to-end (stale override → refused; explicit org default and inherit → queued) — the unit test alone survived swapping the wiring's arguments.

Intended / documented, not changed

  • Fail-closed on picker change: pre-existing scope overrides outside a newly configured picker refuse web turns until re-picked or inherited — that's the point of the fix, but note it when configuring a picker on a live deployment. The picker is one flat org-wide list across harnesses; include ids for every approved harness you want usable on web.
  • Cron/automation turns don't consult the web-UI picker (it's a web knob; Slack is likewise exempt). Post-fix, users can no longer create excluded overrides, so this only affects overrides that predate the picker.
  • With no picker configured, PUT /v1/runtime-config still accepts any approved-harness-supported model (pre-existing; configuring the picker is the boundary the feature promises).

Re-verified live against a running dev instance after the changes: stale override → 403, PUT excluded → 400, explicit excluded → 403, explicit org default → 202, override-to-org-default → 200, inherit → 202.

@16francej
16francej merged commit 2639c85 into main Jul 31, 2026
20 checks passed
@16francej
16francej deleted the playground-deploy-docs branch July 31, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant