Document playground deployments in the deployment workflow - #39
Merged
Conversation
The portal README documents the mechanism, but the deployment workflow never mentioned playground mode, so an operator following the deploy path could not discover it or the companion settings a public try-it instance needs: its own dedicated deployment, an org-wide spend ceiling, per-principal brakes, and a pinned model.
The webui-models picker only gated the explicit per-turn model option. PUT /v1/runtime-config accepted any model an approved harness supports, so a scope override could pin a model the admin had excluded, and turns then ran it without re-validation. The route now refuses a model outside the configured picker, and web turn resolution refuses a scope override outside it as well, so overrides that predate a picker change fail closed instead of running. The org default stays exempt: the picker semantics already treat it as the default selection.
…ides The web UI always sends an explicit model — the default selection is the org base model — so a picker that excluded the org default refused every default turn from the real client. The enabled set is now the configured picker plus the org default, the runtime-config route accepts the org default as an override, and the keep-branch legacy migration stops minting selections the picker excludes. Adds an app-level test pinning the stale-override refusal and both exemption paths, which the unit test alone could not: it survived swapping the wiring's arguments.
Contributor
Author
|
Fresh-context adversarial review of the picker enforcement ran; disposition: Fixed
Intended / documented, not changed
Re-verified live against a running dev instance after the changes: stale override → 403, PUT excluded → 400, explicit excluded → 403, explicit org default → 202, override-to-org-default → 200, inherit → 202. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #38, which added playground mode but documented it only in the portal README — an operator following the deploy path could never discover it.
Adds a "Playground mode" subsection to step 3 of the deployment workflow (
cli/templates/deployment/deployment.md, materialized into every org's deployment directory byqm init), covering:env.portal.PORTAL_PLAYGROUNDstack snippetORG_BUDGET_USD_PER_WINDOWas the hard ceiling, per-principalBUDGET_USD_PER_WINDOW/RATE_LIMIT_PER_WINDOW, and a pinned model via the Admin page's base-model / web-UI model resourcesDocs only; CLI init tests pass (17/17).
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Why the CLI version bump
cli/templates/deployment/deployment.mdships inside the@yc-software/qmnpm package (it's whatqm initmaterializes into a deployment repo). CI's "CLI version bump" check refuses any change to packaged files without a version bump past the published0.1.2, so this PR bumpscli/package.jsonto0.1.3. Doc template change only — no CLI code changes ride along.Also: enforce the web-UI model picker at every core entry
Reviewing the docs claim surfaced a bypass:
webui-modelsonly gated the explicit per-turnmodel:option.PUT /v1/runtime-configaccepted any model an approved harness supports, so a user could pin an excluded model as a scope override and turns ran it without re-validation.Fixed at both entries: the route now refuses a model outside the configured picker (
400 model_not_enabled), and web turn resolution refuses a resolved scope override outside it (covers overrides that predate a picker change). The org default stays exempt — the picker already treats it as the default selection. Unconfigured picker keeps current behavior.Verified live against a running dev instance: stale override → 403 refused; PUT excluded model → 400; explicit per-turn excluded model → 403; org default turn → 202 queued. Plus unit/integration tests (
turn-options,admin-resources).