From 763e66fa770ff3ef600615fa5000359d3908decd Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 02:02:09 +0400 Subject: [PATCH 1/8] Promote each staging deployment's tested entrypoint revision The main deployment's beta job and the Stage 12 comparison path's staging deployment now pass promote_entrypoint: true, as production deployments do. Each staging deployment therefore moves the staging service's stable URL to the revision it tested, runs the same stable-URL checks as production, restores the previous revision if they fail, and then deletes the staging service's older revisions except the promoted one and the one it replaced. The new rule sim.deploy.staging-promotes-tested-revision requires it: a test reads every workflow, finds each job that runs the reusable deployment for staging, and fails unless it passes promote_entrypoint: true. Its proof fixtures are the beta job before and after this change. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/simulation-deploy.yml | 5 +- .github/workflows/stage12-deploy.yml | 4 +- AGENTS.md | 2 +- README.md | 20 ++-- changelog_entry.yaml | 1 + docs/generated/rules.md | 7 +- .../DEPLOYMENT.md | 44 +++++---- .../policyengine-simulation-entry/README.md | 5 +- .../tests/test_deployment_assets.py | 91 ++++++++++++++++++- .../bad_simulation_deploy.yml | 44 +++++++++ .../good_simulation_deploy.yml | 43 +++++++++ rules/registry.toml | 25 +++++ 12 files changed, 259 insertions(+), 32 deletions(-) create mode 100644 rules/fixtures/sim.deploy.staging-promotes-tested-revision/bad_simulation_deploy.yml create mode 100644 rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml diff --git a/.github/workflows/simulation-deploy.yml b/.github/workflows/simulation-deploy.yml index ab3d6d6f4..237962fa0 100644 --- a/.github/workflows/simulation-deploy.yml +++ b/.github/workflows/simulation-deploy.yml @@ -40,7 +40,10 @@ jobs: entrypoint_revision_prefix: s- entrypoint_min_instances: 0 entrypoint_max_instances: 2 - promote_entrypoint: false + # Staging's stable URL must serve the revision this deployment tested, + # because staging callers such as the Public API's staging service use + # that URL. Rule: sim.deploy.staging-promotes-tested-revision. + promote_entrypoint: true run_full_integration: true deploy_existing_stack: true deploy_stage12_v2: true diff --git a/.github/workflows/stage12-deploy.yml b/.github/workflows/stage12-deploy.yml index cecb6b097..35fa719e3 100644 --- a/.github/workflows/stage12-deploy.yml +++ b/.github/workflows/stage12-deploy.yml @@ -53,7 +53,9 @@ jobs: entrypoint_revision_prefix: s- entrypoint_min_instances: 0 entrypoint_max_instances: 2 - promote_entrypoint: false + # Like the main deployment, promote the tested staging revision. + # Rule: sim.deploy.staging-promotes-tested-revision. + promote_entrypoint: true # This workflow deploys and validates only the independent Stage 12 path. # The main API deployment continues to run the existing-stack suite. run_full_integration: false diff --git a/AGENTS.md b/AGENTS.md index 498175b75..1c5a1827d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -114,7 +114,7 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`. ### Deployment pipeline -- Checked: `sim.deploy.container-context-minimal`, `sim.deploy.docker-images-build`, `sim.deploy.gcloud-without-terraform`, `sim.deploy.modal-image-smoke`, `sim.deploy.numerical-sentinel-gate`, `sim.deploy.permission-enforcement-wired`, `sim.deploy.promotion-order-explicit`, `sim.deploy.release-identity-smoke`, `sim.deploy.revision-tags-valid`, `sim.deploy.routing-publication-journaled`, `sim.deploy.shell-scripts-valid`, `sim.deploy.smoke-tests-strict`, `sim.deploy.traffic-changes-exact`, `sim.deploy.workflow-shell-safety` +- Checked: `sim.deploy.container-context-minimal`, `sim.deploy.docker-images-build`, `sim.deploy.gcloud-without-terraform`, `sim.deploy.modal-image-smoke`, `sim.deploy.numerical-sentinel-gate`, `sim.deploy.permission-enforcement-wired`, `sim.deploy.promotion-order-explicit`, `sim.deploy.release-identity-smoke`, `sim.deploy.revision-tags-valid`, `sim.deploy.routing-publication-journaled`, `sim.deploy.shell-scripts-valid`, `sim.deploy.smoke-tests-strict`, `sim.deploy.staging-promotes-tested-revision`, `sim.deploy.traffic-changes-exact`, `sim.deploy.workflow-shell-safety` - `sim.deploy.image-smoke-covers-every-image-lib`: The image smoke path filter includes every library mounted into a Modal image, so a change to any of them runs the smoke. (doc-only, not checked: follow it yourself) ### API contracts and integration diff --git a/README.md b/README.md index 3a3b8dcd8..c620c529e 100644 --- a/README.md +++ b/README.md @@ -91,13 +91,19 @@ time: three parallel service jobs; 3. publish routing only after all three service jobs pass; 4. run generated-client and live authentication tests through the complete - three-service request path; and -5. begin the `prod` deployment only after every `beta` deployment and - qualification job succeeds; -6. assign stable production traffic to the exact tested entrypoint revision - after the complete production suite passes; and -7. verify the stable endpoint, restoring its previous revision if immediate - post-promotion checks fail. + three-service request path; +5. assign the environment's stable traffic to the exact tested entrypoint + revision after that environment's tests pass, in `beta` as in `prod`; +6. verify the stable endpoint, restoring its previous revision if immediate + post-promotion checks fail; and +7. begin the `prod` deployment only after every `beta` deployment, + qualification and promotion job succeeds. + +The staging service's stable URL therefore serves the revision that passed the +staging tests, so staging callers of that URL, such as the Public API's staging +service, reach the build the deployment tested. After each successful +promotion, the deployment deletes the service's older revisions, keeping only +the promoted revision and the one it replaced, for rollback. A successful push-triggered full-stack deployment publishes the API client from the exact deployed commit. diff --git a/changelog_entry.yaml b/changelog_entry.yaml index 2da3c0513..c59857a31 100644 --- a/changelog_entry.yaml +++ b/changelog_entry.yaml @@ -10,6 +10,7 @@ - Ran Stage 12 US national simulations as 20 region-group calls each, and built an independent, immutable Stage 12 cache of three annual US datasets and 60 current-law baselines before each deployment, publishing it only after a recorded determinism check and recording its release in the v2 version manifest under a new key, while the existing key keeps a copy that earlier readers accept - Made the budget refuse a household count outside its country's plausible range (above 200 million in the US or 60 million in the UK, negative or not finite), raising ImplausibleEntityCountError instead of publishing a count like #767's changed: + - Promoted each staging deployment's tested Simulation Entrypoint revision to the staging service's stable traffic, as production deployments already do, in both the main deployment's beta job and the Stage 12 comparison path's staging deployment; the staging stable URL now serves the build that passed the staging tests, promotion runs the same stable-URL checks as production's and restores the previous revision when they fail, and the existing clean-up then deletes the staging service's older revisions except the promoted one and the one it replaced; the new rule sim.deploy.staging-promotes-tested-revision requires it - Enabled every ruff rule (select ALL), ignoring only COM812 and ISC001, which conflict with ruff format, and the D203 and D213 halves of the mutually exclusive docstring pairs; complex functions, long parameter lists and magic numbers were split into smaller helpers and grouped parameters, each behind new characterization tests, across the contract and persistence validation, entry settings, app factory and Stage 12 adapter, gateway route resolution, Stage 12 coordination, workers, parity and qualification, the precompute and national reduce, the routing publisher and its command line, the release sentinel, the package update helper and the shared rules tool; Modal function, FastAPI route, Pydantic validator and command-line signatures and every OpenAPI document are unchanged; the contract library's batch-state builder and SPM selection resolver take their route values as one grouped argument, the simulation observability initializer drops a service_version parameter no caller passed, and no PLR0913 suppression is needed - Enabled ruff's argument, idiom and pytest-style rules (ARG, ASYNC, B, C4, FBT, PD, PERF, PGH, PLW, PT, PTH, RUF, SIM); unused arguments are removed, underscored or explained, FastAPI auth dependencies use Annotated, re-raised errors keep their cause, and boolean options of in-repo helpers are keyword-only; OpenAPI documents are unchanged, and the shared rules tool's unittest tests are exempt from pytest-style rules; tests take the unused-argument rules (ARG001, ARG002, ARG005) as file-level exceptions, since their fakes and stubs mirror real signatures; best-effort telemetry and cleanup calls that swallow any error use contextlib.suppress, with the reason as a comment - Enabled ruff's structure and import rules (INP, PLC, SLF, N, A, FAST, E); long lines are wrapped, deferred imports that Modal images, test seams, import cycles or heavy dependencies need carry a reason, script directories are declared to ruff as namespace packages, four internal exceptions gained an Error suffix, and published field names, OpenAPI documents and logged error types are unchanged; tests are also exempt from SLF001, INP001 and PLC0415; heavy or optional packages that load only inside functions are listed in ruff's banned-module-level-imports, which keeps them out of module-level imports (TID253) and lets their deferred imports go without a suppression, and the Modal utility programs take INP001 file-wide diff --git a/docs/generated/rules.md b/docs/generated/rules.md index 193ef9873..1ca667aac 100644 --- a/docs/generated/rules.md +++ b/docs/generated/rules.md @@ -8,11 +8,11 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf | Tier | enforced | ratcheted | advisory | doc-only | Total | | --- | ---: | ---: | ---: | ---: | ---: | -| mechanical | 49 | 5 | 0 | 15 | 69 | +| mechanical | 50 | 5 | 0 | 15 | 70 | | metric | 0 | 0 | 0 | 0 | 0 | | judgment | 0 | 0 | 0 | 0 | 0 | -**Blocking:** 54 of 69 rules. +**Blocking:** 55 of 70 rules. ## Gaps @@ -155,6 +155,7 @@ flowchart LR | `sim.deploy.routing-publication-journaled` | Each routing publication is named by its run, attempt and environment and journaled before it is applied; an always-running finalizer resolves it from the journal with the release-gate outcome, a release reports success only after that finalizer completes, and routing jobs never tolerate a failed command. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | | `sim.deploy.shell-scripts-valid` | Deployment shell scripts have valid shell syntax. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | | `sim.deploy.smoke-tests-strict` | Candidate and stable smoke tests reject a revision mismatch without retrying, and fail if the previous revision does not converge. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | +| `sim.deploy.staging-promotes-tested-revision` | Every staging deployment passes promote_entrypoint: true, as production deployments do, so the staging service's stable URL serves the entrypoint revision that passed the staging tests. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | yes | | `sim.deploy.traffic-changes-exact` | Traffic changes target an exact, validated revision, are reversible, and refuse to run over an intervening promotion. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | | `sim.deploy.workflow-shell-safety` | Cloud Run deployment never interpolates GitHub expressions into shell, escapes environment values, and pins secret versions. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | @@ -180,6 +181,7 @@ flowchart LR r_sim_deploy_routing_publication_journaled("sim.deploy.routing-publication-journaled (enforced)") r_sim_deploy_shell_scripts_valid("sim.deploy.shell-scripts-valid (enforced)") r_sim_deploy_smoke_tests_strict("sim.deploy.smoke-tests-strict (enforced)") + r_sim_deploy_staging_promotes_tested_revision("sim.deploy.staging-promotes-tested-revision (enforced)") r_sim_deploy_traffic_changes_exact("sim.deploy.traffic-changes-exact (enforced)") r_sim_deploy_workflow_shell_safety("sim.deploy.workflow-shell-safety (enforced)") r_sim_deploy_container_context_minimal --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py @@ -198,6 +200,7 @@ flowchart LR r_sim_deploy_routing_publication_journaled --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py r_sim_deploy_shell_scripts_valid --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py r_sim_deploy_smoke_tests_strict --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py + r_sim_deploy_staging_promotes_tested_revision --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py r_sim_deploy_traffic_changes_exact --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py r_sim_deploy_workflow_shell_safety --> e_pytest_projects_policyengine_simulation_entry_tests_test_deployment_assets_py ``` diff --git a/projects/policyengine-simulation-entry/DEPLOYMENT.md b/projects/policyengine-simulation-entry/DEPLOYMENT.md index 9783ef1cc..7a7bfb591 100644 --- a/projects/policyengine-simulation-entry/DEPLOYMENT.md +++ b/projects/policyengine-simulation-entry/DEPLOYMENT.md @@ -33,17 +33,24 @@ qualification uses Cloud Run's generated tagged revision URL and does not require a custom domain. The `prod` environment follows the same sequence automatically, but cannot -begin until every `beta` deployment, routing, integration, and authentication -job has succeeded. Production tests also use the tagged candidate URL, so the -currently serving revision remains unchanged during qualification. +begin until every `beta` deployment, routing, integration, authentication, and +promotion job has succeeded. Both environments' tests use the tagged candidate +URL, so the currently serving revision remains unchanged during qualification. ## Promotion and rollback -After the complete production suite passes, the workflow verifies that the -exact tested revision belongs to the production service and is ready. It also -checks that stable traffic still points to the revision recorded before -deployment. If those guards pass, it assigns all stable service traffic to the -tested production revision. +Both `beta` and `prod` promote. After an environment's complete suite passes, +the workflow verifies that the exact tested revision belongs to that +environment's service and is ready. It also checks that stable traffic still +points to the revision recorded before deployment. If those checks pass, it +assigns all stable service traffic to the tested revision. + +In `beta`, promotion keeps the staging service's stable URL on the revision +that passed the staging tests, so staging callers of that URL, such as the +Public API's staging service, reach the build the deployment tested. The +staging deployment of the **Deploy Stage 12 comparison path** workflow promotes +in the same way. Rule `sim.deploy.staging-promotes-tested-revision` requires +both. The workflow then checks the stable generated Cloud Run URL. It uses the runtime revision header to prove that the stable URL reaches the exact revision @@ -51,6 +58,10 @@ that passed candidate tests, and it repeats the deployed authentication checks. If an immediate post-promotion check fails, the workflow restores the exact previous revision recorded before deployment and fails visibly. +After a successful promotion and deployment, the workflow deletes the +service's older revisions. It keeps only the promoted revision and the one it +replaced, which `simulation-rollback.yml` can route back to. + Deployment concurrency prevents two releases from changing the entrypoint service simultaneously. Promotion never targets `LATEST`. @@ -82,9 +93,9 @@ The deployment workflow passes each environment's unset or empty variable deploys `0`, which records callers that lack a route's permission but lets them through. `1` rejects those callers with 403. Any other value fails the deployment before Cloud Run changes. A new value takes effect -at the next entrypoint deployment. In `prod` that deployment is promoted to -stable traffic. `beta` never promotes its candidate revision, so a `beta` value -applies only to the tagged candidate that the deployment tests call. The route +at the next entrypoint deployment, which both `beta` and `prod` promote to +stable traffic once their tests pass. A `beta` value therefore applies to every +caller of the staging stable URL as well as to the deployment tests. The route permissions are listed in `README.md`. ### Auth0 prerequisites @@ -117,11 +128,12 @@ variable to `1`: 1. Deploy with the variable unset or `0`, and complete the Auth0 setup. 2. Watch the `prod` service for `simulation_entry_permission_missing` events with `enforced=false`. Each one names the missing permission and the - caller's `client_id`. Only `prod` serves real callers such as API v1: a - `beta` candidate receives only the deployment tests' own requests, so `beta` - events cannot show a missing API v1 grant. Access tokens issued before a - grant do not carry it, so wait at least one access-token lifetime after the - last grant with no new events. + caller's `client_id`. Only `prod` is certain to receive requests from every + real caller: the staging stable URL receives requests only from the staging + callers configured to use it, and tagged candidates only from the + deployment tests, so `beta` events cannot show every missing grant. Access + tokens issued before a grant do not carry it, so wait at least one + access-token lifetime after the last grant with no new events. 3. Set the `beta` variable to `1` and deploy. With enforcement on, the authenticated deployment tests also check that the test client's token is rejected on `GET /internal/stage12/reports/{evaluation_id}`, which requires diff --git a/projects/policyengine-simulation-entry/README.md b/projects/policyengine-simulation-entry/README.md index 3a3f72815..272f2cbf0 100644 --- a/projects/policyengine-simulation-entry/README.md +++ b/projects/policyengine-simulation-entry/README.md @@ -21,8 +21,9 @@ Cloud Run's generated revision name is returned in a response header so deployment checks can prove which revision a tagged or stable URL serves. Live caller/backend authentication checks are isolated in `authenticated_tests`. -They qualify tagged Cloud Run candidates before production traffic changes and -verify the stable production endpoint immediately after promotion. +They qualify tagged Cloud Run candidates before an environment's stable traffic +changes and verify its stable endpoint immediately after promotion, in both +staging and production. ## Caller authentication and permissions diff --git a/projects/policyengine-simulation-entry/tests/test_deployment_assets.py b/projects/policyengine-simulation-entry/tests/test_deployment_assets.py index de151fae2..320dc145d 100644 --- a/projects/policyengine-simulation-entry/tests/test_deployment_assets.py +++ b/projects/policyengine-simulation-entry/tests/test_deployment_assets.py @@ -30,10 +30,20 @@ DEPLOYMENT_READY_SCRIPT = ( REPOSITORY_ROOT / ".github" / "scripts" / "simulation-deployment-ready.sh" ) +WORKFLOWS_DIRECTORY = REPOSITORY_ROOT / ".github" / "workflows" DEPLOY_WORKFLOW = REPOSITORY_ROOT / ".github" / "workflows" / "simulation-deploy.yml" REUSABLE_DEPLOY_WORKFLOW = ( REPOSITORY_ROOT / ".github" / "workflows" / "simulation-deploy.reusable.yml" ) +REUSABLE_DEPLOY_WORKFLOW_REFERENCE = ( + "./.github/workflows/simulation-deploy.reusable.yml" +) +STAGING_PROMOTION_FIXTURES = ( + REPOSITORY_ROOT + / "rules" + / "fixtures" + / "sim.deploy.staging-promotes-tested-revision" +) ROLLBACK_WORKFLOW = ( REPOSITORY_ROOT / ".github" / "workflows" / "simulation-rollback.yml" ) @@ -341,8 +351,8 @@ def _assert_beta_then_prod_release( assert "deployment_environment: staging" in deploy_workflow assert "deployment_environment: production" in deploy_workflow assert deploy_workflow.count("deploy_existing_stack: true") == 2 - assert "promote_entrypoint: false" in deploy_workflow - assert "promote_entrypoint: true" in deploy_workflow + assert deploy_workflow.count("promote_entrypoint: true") == 2 + assert "promote_entrypoint: false" not in deploy_workflow assert "entrypoint_environment" not in deploy_workflow assert "entrypoint_environment" not in reusable_workflow assert "entrypoint_public_url" not in deploy_workflow @@ -667,6 +677,83 @@ def test_revision_tags_fit_existing_cloud_run_service_names(): assert len(service) + len(f"{prefix}9999999") <= 46 +def _staging_deployment_job_ids(jobs: dict) -> list[str]: + """Return the jobs that run the reusable deployment workflow for staging.""" + return [ + job_id + for job_id, job in jobs.items() + if job.get("uses") == REUSABLE_DEPLOY_WORKFLOW_REFERENCE + and (job.get("with") or {}).get("deployment_environment") == "staging" + ] + + +def staging_promotion_violations(jobs: dict) -> list[str]: + """Return each staging deployment job that leaves stable traffic unchanged. + + Rule sim.deploy.staging-promotes-tested-revision: a staging deployment must + pass ``promote_entrypoint: true`` to the reusable deployment workflow, as + production deployments do. Any other value, including an expression, + counts as a violation. + """ + return [ + f"{job_id} must pass promote_entrypoint: true" + for job_id in _staging_deployment_job_ids(jobs) + if jobs[job_id]["with"].get("promote_entrypoint") is not True + ] + + +def test_staging_deployments_promote_their_tested_entrypoint_revision(): + """Staging's stable URL serves the revision that passed staging's tests. + + Callers such as the Public API's staging service use the staging stable + URL, so a staging deployment that left stable traffic on an older revision + would leave them calling a build that the deployment did not test. + """ + staging_jobs_by_workflow = {} + violations = [] + for workflow_path in sorted(WORKFLOWS_DIRECTORY.glob("*.yml")): + jobs = _workflow_jobs(workflow_path) + staging_job_ids = _staging_deployment_job_ids(jobs) + if staging_job_ids: + staging_jobs_by_workflow[workflow_path.name] = staging_job_ids + violations.extend( + f"{workflow_path.name}: {violation}" + for violation in staging_promotion_violations(jobs) + ) + + # The check must find the two known staging deployments, so that a renamed + # input or reusable workflow cannot make it pass by finding nothing. + assert "beta" in staging_jobs_by_workflow.get(DEPLOY_WORKFLOW.name, []) + assert "deploy-stage12-staging" in staging_jobs_by_workflow.get( + STAGE12_DEPLOY_WORKFLOW.name, [] + ) + for workflow_name, staging_job_ids in staging_jobs_by_workflow.items(): + jobs = _workflow_jobs(WORKFLOWS_DIRECTORY / workflow_name) + for job_id in staging_job_ids: + assert ( + jobs[job_id]["with"]["entrypoint_service"] + == "policyengine-simulation-entry-staging" + ) + assert violations == [] + + +def test_rule_proof_staging_promotes_tested_revision(): + """Proof for rule sim.deploy.staging-promotes-tested-revision. + + The bad fixture is the `beta` deployment as it was before the rule: it + tested a tagged candidate but never moved staging's stable traffic to it. + """ + bad_jobs = _workflow_jobs(STAGING_PROMOTION_FIXTURES / "bad_simulation_deploy.yml") + good_jobs = _workflow_jobs( + STAGING_PROMOTION_FIXTURES / "good_simulation_deploy.yml" + ) + + assert staging_promotion_violations(bad_jobs) == [ + "beta must pass promote_entrypoint: true" + ] + assert staging_promotion_violations(good_jobs) == [] + + def test_entrypoint_container_uses_requested_target_platform(): dockerfile = DOCKERFILE.read_text(encoding="utf-8") diff --git a/rules/fixtures/sim.deploy.staging-promotes-tested-revision/bad_simulation_deploy.yml b/rules/fixtures/sim.deploy.staging-promotes-tested-revision/bad_simulation_deploy.yml new file mode 100644 index 000000000..fbff8dc6a --- /dev/null +++ b/rules/fixtures/sim.deploy.staging-promotes-tested-revision/bad_simulation_deploy.yml @@ -0,0 +1,44 @@ +# Proof fixture: the deployment workflow's jobs as they were before the rule. +# "beta" deploys and tests a tagged staging candidate but passes +# promote_entrypoint: false, so the staging stable URL keeps serving whatever +# revision it served before, which the deployment never tested. +name: Deploy Simulation API + +on: + push: + branches: + - main + +jobs: + beta: + name: Complete beta deployment + uses: ./.github/workflows/simulation-deploy.reusable.yml + with: + release_environment: beta + deployment_environment: staging + modal_environment: staging + entrypoint_service: policyengine-simulation-entry-staging + entrypoint_revision_prefix: s- + entrypoint_min_instances: 0 + entrypoint_max_instances: 2 + promote_entrypoint: false + run_full_integration: true + deploy_existing_stack: true + secrets: inherit + + prod: + name: Complete prod deployment + needs: beta + uses: ./.github/workflows/simulation-deploy.reusable.yml + with: + release_environment: prod + deployment_environment: production + modal_environment: main + entrypoint_service: policyengine-simulation-entry + entrypoint_revision_prefix: p- + entrypoint_min_instances: 1 + entrypoint_max_instances: 20 + promote_entrypoint: true + run_full_integration: false + deploy_existing_stack: true + secrets: inherit diff --git a/rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml b/rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml new file mode 100644 index 000000000..a8f90d276 --- /dev/null +++ b/rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml @@ -0,0 +1,43 @@ +# Proof fixture: the deployment workflow's jobs with the rule satisfied. +# "beta" promotes its tested staging candidate, as "prod" does, so the staging +# stable URL serves the revision that passed the staging tests. +name: Deploy Simulation API + +on: + push: + branches: + - main + +jobs: + beta: + name: Complete beta deployment + uses: ./.github/workflows/simulation-deploy.reusable.yml + with: + release_environment: beta + deployment_environment: staging + modal_environment: staging + entrypoint_service: policyengine-simulation-entry-staging + entrypoint_revision_prefix: s- + entrypoint_min_instances: 0 + entrypoint_max_instances: 2 + promote_entrypoint: true + run_full_integration: true + deploy_existing_stack: true + secrets: inherit + + prod: + name: Complete prod deployment + needs: beta + uses: ./.github/workflows/simulation-deploy.reusable.yml + with: + release_environment: prod + deployment_environment: production + modal_environment: main + entrypoint_service: policyengine-simulation-entry + entrypoint_revision_prefix: p- + entrypoint_min_instances: 1 + entrypoint_max_instances: 20 + promote_entrypoint: true + run_full_integration: false + deploy_existing_stack: true + secrets: inherit diff --git a/rules/registry.toml b/rules/registry.toml index c0308225f..d7b1e4664 100644 --- a/rules/registry.toml +++ b/rules/registry.toml @@ -375,6 +375,31 @@ workflow = ".github/workflows/pr.yml" job = "test" invocation = "uv run --frozen pytest tests/" +[[rule]] +id = "sim.deploy.staging-promotes-tested-revision" +category = "deployment" +statement = "Every staging deployment passes promote_entrypoint: true, as production deployments do, so the staging service's stable URL serves the entrypoint revision that passed the staging tests." +rationale = "projects/policyengine-simulation-entry/DEPLOYMENT.md#promotion-and-rollback" +scope = [".github/workflows/*.yml"] +tier = "mechanical" +status = "enforced" +autofix = false +[rule.enforcer] +kind = "pytest" +project = "projects/policyengine-simulation-entry" +nodes = [ + "tests/test_deployment_assets.py::test_staging_deployments_promote_their_tested_entrypoint_revision", +] +[[rule.ci]] +workflow = ".github/workflows/pr.yml" +job = "test" +invocation = "uv run --frozen pytest tests/" +[rule.proof] +bad_fixture = "rules/fixtures/sim.deploy.staging-promotes-tested-revision/bad_simulation_deploy.yml" +good_fixture = "rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml" +project = "projects/policyengine-simulation-entry" +test = "tests/test_deployment_assets.py::test_rule_proof_staging_promotes_tested_revision" + [[rule]] id = "sim.deploy.smoke-tests-strict" category = "deployment" From 2b7cde8192078eda1d22762d8747780dc9984c10 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 02:04:26 +0400 Subject: [PATCH 2/8] Document that a manual staging run from another branch promotes that build The beta environment does not restrict which branch a run may come from, so a manual run of either deployment workflow from a branch other than main now moves the staging stable URL to that branch's build until the next deployment from main. Co-Authored-By: Claude Opus 5.5 --- projects/policyengine-simulation-entry/DEPLOYMENT.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/projects/policyengine-simulation-entry/DEPLOYMENT.md b/projects/policyengine-simulation-entry/DEPLOYMENT.md index 7a7bfb591..c9ec55e26 100644 --- a/projects/policyengine-simulation-entry/DEPLOYMENT.md +++ b/projects/policyengine-simulation-entry/DEPLOYMENT.md @@ -52,6 +52,13 @@ staging deployment of the **Deploy Stage 12 comparison path** workflow promotes in the same way. Rule `sim.deploy.staging-promotes-tested-revision` requires both. +Both workflows can also be started manually, and the `beta` environment does +not restrict which branch a run may come from. A manual run from a branch other +than `main` therefore promotes that branch's build to the staging stable URL, +and every staging caller of that URL reaches the unmerged build until the next +deployment from `main` promotes again. Start a staging deployment from another +branch only when that is intended. + The workflow then checks the stable generated Cloud Run URL. It uses the runtime revision header to prove that the stable URL reaches the exact revision that passed candidate tests, and it repeats the deployed authentication checks. From e9e8de97070bb74781a428cda7554a864cb6452e Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 02:06:18 +0400 Subject: [PATCH 3/8] Document that production waits for the staging revision clean-up The prod deployment needs the whole beta deployment, which now includes the revision clean-up that runs after a staging promotion, so a failed staging clean-up also stops the production deployment. Co-Authored-By: Claude Opus 5.5 --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c620c529e..6cd9cec12 100644 --- a/README.md +++ b/README.md @@ -97,7 +97,8 @@ time: 6. verify the stable endpoint, restoring its previous revision if immediate post-promotion checks fail; and 7. begin the `prod` deployment only after every `beta` deployment, - qualification and promotion job succeeds. + qualification, promotion and revision clean-up job succeeds, so a failed + staging clean-up also stops the production deployment. The staging service's stable URL therefore serves the revision that passed the staging tests, so staging callers of that URL, such as the Public API's staging From 4f34bf0f5b88e23a62b2b225ddf30a6109c8b0d0 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Thu, 22 Oct 2026 17:10:38 +0400 Subject: [PATCH 4/8] State in the entrypoint deployment guide that production waits for the staging clean-up The repository README already says the prod deployment starts only after beta's revision clean-up succeeds; the entrypoint's DEPLOYMENT.md listed the beta jobs prod waits for without the clean-up job. Co-Authored-By: Claude Opus 5.5 --- projects/policyengine-simulation-entry/DEPLOYMENT.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/projects/policyengine-simulation-entry/DEPLOYMENT.md b/projects/policyengine-simulation-entry/DEPLOYMENT.md index c9ec55e26..fbb248291 100644 --- a/projects/policyengine-simulation-entry/DEPLOYMENT.md +++ b/projects/policyengine-simulation-entry/DEPLOYMENT.md @@ -33,9 +33,11 @@ qualification uses Cloud Run's generated tagged revision URL and does not require a custom domain. The `prod` environment follows the same sequence automatically, but cannot -begin until every `beta` deployment, routing, integration, authentication, and -promotion job has succeeded. Both environments' tests use the tagged candidate -URL, so the currently serving revision remains unchanged during qualification. +begin until every `beta` deployment, routing, integration, authentication, +promotion, and revision clean-up job has succeeded, so a failed staging +clean-up also stops the production deployment. Both environments' tests use +the tagged candidate URL, so the currently serving revision remains unchanged +during qualification. ## Promotion and rollback From 0501a0f4e29ca9bc5acc4b15953aad00ac128708 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Fri, 30 Oct 2026 20:33:59 +0400 Subject: [PATCH 5/8] Add failing tests for promoting staging only from main The staging promotion rule now requires each staging deployment to pass promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}. A literal true counts as a violation, because a manual run from another branch would then move the staging stable URL to an unmerged build, and the clean-up would delete the older revisions that rollback needs. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_deployment_assets.py | 6 ++- .../tests/test_deployment_assets_promotion.py | 41 +++++++++++++++---- .../good_simulation_deploy.yml | 7 ++-- 3 files changed, 41 insertions(+), 13 deletions(-) diff --git a/projects/policyengine-simulation-entry/tests/test_deployment_assets.py b/projects/policyengine-simulation-entry/tests/test_deployment_assets.py index 975350483..503b30295 100644 --- a/projects/policyengine-simulation-entry/tests/test_deployment_assets.py +++ b/projects/policyengine-simulation-entry/tests/test_deployment_assets.py @@ -198,7 +198,11 @@ def _assert_beta_then_prod_release( assert "deployment_environment: staging" in deploy_workflow assert "deployment_environment: production" in deploy_workflow assert deploy_workflow.count("deploy_existing_stack: true") == 2 - assert deploy_workflow.count("promote_entrypoint: true") == 2 + # prod always promotes; beta promotes only a run from main. + assert deploy_workflow.count("promote_entrypoint: true") == 1 + assert ( + "promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}" in deploy_workflow + ) assert "promote_entrypoint: false" not in deploy_workflow assert "entrypoint_environment" not in deploy_workflow assert "entrypoint_environment" not in reusable_workflow diff --git a/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py b/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py index 124f8c49c..4295e062a 100644 --- a/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py +++ b/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py @@ -2,6 +2,7 @@ from __future__ import annotations +import copy import re from test_deployment_assets_support import ( @@ -12,6 +13,7 @@ STAGE12_DEPLOY_WORKFLOW, STAGING_PROMOTION_FIXTURES, WORKFLOWS_DIRECTORY, + evaluate_github_expression, job_section, sentinel_check_step, workflow_jobs, @@ -272,6 +274,12 @@ def test_deployed_stack_must_reproduce_the_numerical_sentinel(): ) +# The promotion input every staging deployment passes: true only for a run +# from main, so a manual run from another branch never moves staging's stable +# traffic or deletes its older revisions. +MAIN_ONLY_PROMOTION = "${{ github.ref == 'refs/heads/main' }}" + + def _staging_deployment_job_ids(jobs: dict) -> list[str]: """Return the jobs that run the reusable deployment workflow for staging.""" return [ @@ -283,17 +291,18 @@ def _staging_deployment_job_ids(jobs: dict) -> list[str]: def staging_promotion_violations(jobs: dict) -> list[str]: - """Return each staging deployment job that leaves stable traffic unchanged. + """Return each staging deployment job that doesn't promote exactly from main. Rule sim.deploy.staging-promotes-tested-revision: a staging deployment must - pass ``promote_entrypoint: true`` to the reusable deployment workflow, as - production deployments do. Any other value, including an expression, - counts as a violation. + pass ``promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}`` to the + reusable deployment workflow. Any other value counts as a violation: + ``false`` leaves the stable URL on a revision the deployment never tested, + and ``true`` lets a run from another branch promote an unmerged build. """ return [ - f"{job_id} must pass promote_entrypoint: true" + f"{job_id} must pass promote_entrypoint: {MAIN_ONLY_PROMOTION}" for job_id in _staging_deployment_job_ids(jobs) - if jobs[job_id]["with"].get("promote_entrypoint") is not True + if jobs[job_id]["with"].get("promote_entrypoint") != MAIN_ONLY_PROMOTION ] @@ -337,11 +346,25 @@ def test_rule_proof_staging_promotes_tested_revision(): The bad fixture is the `beta` deployment as it was before the rule: it tested a tagged candidate but never moved staging's stable traffic to it. + Passing a literal ``true`` is also a violation, since a run from another + branch would then promote that branch's build. """ bad_jobs = workflow_jobs(STAGING_PROMOTION_FIXTURES / "bad_simulation_deploy.yml") good_jobs = workflow_jobs(STAGING_PROMOTION_FIXTURES / "good_simulation_deploy.yml") + any_branch_jobs = copy.deepcopy(good_jobs) + any_branch_jobs["beta"]["with"]["promote_entrypoint"] = True - assert staging_promotion_violations(bad_jobs) == [ - "beta must pass promote_entrypoint: true" - ] + violation = f"beta must pass promote_entrypoint: {MAIN_ONLY_PROMOTION}" + assert staging_promotion_violations(bad_jobs) == [violation] + assert staging_promotion_violations(any_branch_jobs) == [violation] assert staging_promotion_violations(good_jobs) == [] + # The required value is true for a run from main and false for any other + # branch or for a tag. + for ref, promotes in ( + ("refs/heads/main", True), + ("refs/heads/feat/stage12-segmentation-cache", False), + ("refs/heads/main-backport", False), + ("refs/tags/v1.0.0", False), + ): + context = {"github": {"ref": ref}} + assert evaluate_github_expression(MAIN_ONLY_PROMOTION, context) is promotes diff --git a/rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml b/rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml index a8f90d276..2b6a33257 100644 --- a/rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml +++ b/rules/fixtures/sim.deploy.staging-promotes-tested-revision/good_simulation_deploy.yml @@ -1,6 +1,7 @@ # Proof fixture: the deployment workflow's jobs with the rule satisfied. -# "beta" promotes its tested staging candidate, as "prod" does, so the staging -# stable URL serves the revision that passed the staging tests. +# "beta" promotes its tested staging candidate when the run is from main, so the +# staging stable URL serves the revision that passed the staging tests, and a +# manual run from another branch leaves it unchanged. name: Deploy Simulation API on: @@ -20,7 +21,7 @@ jobs: entrypoint_revision_prefix: s- entrypoint_min_instances: 0 entrypoint_max_instances: 2 - promote_entrypoint: true + promote_entrypoint: ${{ github.ref == 'refs/heads/main' }} run_full_integration: true deploy_existing_stack: true secrets: inherit From 62ca5357b046d320b1197e8615e88f34eaf104dc Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Fri, 30 Oct 2026 20:33:59 +0400 Subject: [PATCH 6/8] Promote staging only from main The beta deployment and the Stage 12 comparison path's staging deployment pass promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}. A manual run from another branch still deploys and tests its candidate revision, but leaves the staging stable URL and the older staging revisions unchanged. The rule statement, deployment guide, README and changelog fragment say so, and the rules documents are re-rendered. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/simulation-deploy.yml | 6 ++++-- .github/workflows/stage12-deploy.yml | 6 +++--- README.md | 3 ++- changelog.d/893.changed.md | 2 +- docs/generated/rules.md | 2 +- .../DEPLOYMENT.md | 20 ++++++++++--------- rules/registry.toml | 2 +- 7 files changed, 23 insertions(+), 18 deletions(-) diff --git a/.github/workflows/simulation-deploy.yml b/.github/workflows/simulation-deploy.yml index c9b854da0..a4f33808f 100644 --- a/.github/workflows/simulation-deploy.yml +++ b/.github/workflows/simulation-deploy.yml @@ -101,8 +101,10 @@ jobs: entrypoint_max_instances: 2 # Staging's stable URL must serve the revision this deployment tested, # because staging callers such as the Public API's staging service use - # that URL. Rule: sim.deploy.staging-promotes-tested-revision. - promote_entrypoint: true + # that URL. Only a run from main promotes: a manual run from another + # branch tests its candidate but leaves stable traffic and older + # revisions alone. Rule: sim.deploy.staging-promotes-tested-revision. + promote_entrypoint: ${{ github.ref == 'refs/heads/main' }} run_full_integration: true deploy_existing_stack: true deploy_stage12_v2: true diff --git a/.github/workflows/stage12-deploy.yml b/.github/workflows/stage12-deploy.yml index 35fa719e3..a5b252e3f 100644 --- a/.github/workflows/stage12-deploy.yml +++ b/.github/workflows/stage12-deploy.yml @@ -53,9 +53,9 @@ jobs: entrypoint_revision_prefix: s- entrypoint_min_instances: 0 entrypoint_max_instances: 2 - # Like the main deployment, promote the tested staging revision. - # Rule: sim.deploy.staging-promotes-tested-revision. - promote_entrypoint: true + # Like the main deployment, promote the tested staging revision, and + # only from main. Rule: sim.deploy.staging-promotes-tested-revision. + promote_entrypoint: ${{ github.ref == 'refs/heads/main' }} # This workflow deploys and validates only the independent Stage 12 path. # The main API deployment continues to run the existing-stack suite. run_full_integration: false diff --git a/README.md b/README.md index fc5540298..308e475fb 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,8 @@ push deploys one complete stack at a time (see 4. run generated-client and live authentication tests through the complete three-service request path; 5. assign the environment's stable traffic to the exact tested entrypoint - revision after that environment's tests pass, in `beta` as in `prod`; + revision after that environment's tests pass, in `beta` as in `prod` + (`beta` only for a run from `main`); 6. verify the stable endpoint, restoring its previous revision if immediate post-promotion checks fail; and 7. begin the `prod` deployment only after every `beta` deployment, diff --git a/changelog.d/893.changed.md b/changelog.d/893.changed.md index 38c8773cf..a1a625fad 100644 --- a/changelog.d/893.changed.md +++ b/changelog.d/893.changed.md @@ -1 +1 @@ -Promoted each staging deployment's tested Simulation Entrypoint revision to the staging service's stable traffic, as production deployments already do, in both the main deployment's beta job and the Stage 12 comparison path's staging deployment. The staging stable URL now serves the build that passed the staging tests; promotion runs the same stable-URL checks as production's and restores the previous revision when they fail, and the existing clean-up then deletes the staging service's older revisions except the promoted one and the one it replaced. The new rule sim.deploy.staging-promotes-tested-revision requires it. +Promoted each staging deployment's tested Simulation Entrypoint revision to the staging service's stable traffic, as production deployments already do, in both the main deployment's beta job and the Stage 12 comparison path's staging deployment. The staging stable URL now serves the build that passed the staging tests; promotion runs the same stable-URL checks as production's and restores the previous revision when they fail, and the existing clean-up then deletes the staging service's older revisions except the promoted one and the one it replaced. Only a run from main promotes: a manual staging run from another branch tests its candidate but leaves the stable URL and older revisions unchanged. The new rule sim.deploy.staging-promotes-tested-revision requires it. diff --git a/docs/generated/rules.md b/docs/generated/rules.md index 00c7f9a95..f83479118 100644 --- a/docs/generated/rules.md +++ b/docs/generated/rules.md @@ -156,7 +156,7 @@ flowchart LR | `sim.deploy.routing-publication-journaled` | Each routing publication is named by its run, attempt and environment and journaled before it is applied; an always-running finalizer resolves it from the journal with the release-gate outcome, a release reports success only after that finalizer completes, and routing jobs never tolerate a failed command. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_release_gates.py`, `projects/policyengine-simulation-entry/tests/test_deployment_assets_routing.py` | `pr.yml:test` | **unproven** | | `sim.deploy.shell-scripts-valid` | Deployment shell scripts have valid shell syntax. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | | `sim.deploy.smoke-tests-strict` | Candidate and stable smoke tests reject a revision mismatch without retrying, and fail if the previous revision does not converge. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | -| `sim.deploy.staging-promotes-tested-revision` | Every staging deployment passes promote_entrypoint: true, as production deployments do, so the staging service's stable URL serves the entrypoint revision that passed the staging tests. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py` | `pr.yml:test` | yes | +| `sim.deploy.staging-promotes-tested-revision` | Every staging deployment passes promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}, so a run from main moves the staging service's stable URL to the entrypoint revision that passed the staging tests, and a manual run from another branch never changes it. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py` | `pr.yml:test` | yes | | `sim.deploy.traffic-changes-exact` | Traffic changes target an exact, validated revision, are reversible, and refuse to run over an intervening promotion. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_traffic.py` | `pr.yml:test` | **unproven** | | `sim.deploy.workflow-shell-safety` | Cloud Run deployment never interpolates GitHub expressions into shell, escapes environment values, and pins secret versions. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_cloud_run.py` | `pr.yml:test` | **unproven** | diff --git a/projects/policyengine-simulation-entry/DEPLOYMENT.md b/projects/policyengine-simulation-entry/DEPLOYMENT.md index fbb248291..ec2b4beeb 100644 --- a/projects/policyengine-simulation-entry/DEPLOYMENT.md +++ b/projects/policyengine-simulation-entry/DEPLOYMENT.md @@ -41,7 +41,8 @@ during qualification. ## Promotion and rollback -Both `beta` and `prod` promote. After an environment's complete suite passes, +Both `beta` and `prod` promote, `beta` only for a run from `main`. After an +environment's complete suite passes, the workflow verifies that the exact tested revision belongs to that environment's service and is ready. It also checks that stable traffic still points to the revision recorded before deployment. If those checks pass, it @@ -55,11 +56,11 @@ in the same way. Rule `sim.deploy.staging-promotes-tested-revision` requires both. Both workflows can also be started manually, and the `beta` environment does -not restrict which branch a run may come from. A manual run from a branch other -than `main` therefore promotes that branch's build to the staging stable URL, -and every staging caller of that URL reaches the unmerged build until the next -deployment from `main` promotes again. Start a staging deployment from another -branch only when that is intended. +not restrict which branch a run may come from. Both therefore pass +`promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}`: a manual run from +a branch other than `main` deploys and tests its candidate revision, but never +moves the staging stable URL to it and never deletes older staging revisions, +so the last revision deployed from `main` stays available for rollback. The workflow then checks the stable generated Cloud Run URL. It uses the runtime revision header to prove that the stable URL reaches the exact revision @@ -102,9 +103,10 @@ The deployment workflow passes each environment's unset or empty variable deploys `0`, which records callers that lack a route's permission but lets them through. `1` rejects those callers with 403. Any other value fails the deployment before Cloud Run changes. A new value takes effect -at the next entrypoint deployment, which both `beta` and `prod` promote to -stable traffic once their tests pass. A `beta` value therefore applies to every -caller of the staging stable URL as well as to the deployment tests. The route +at the next entrypoint deployment, which `prod`, and `beta` for a run from +`main`, promote to stable traffic once their tests pass. A `beta` value +therefore applies to every caller of the staging stable URL as well as to the +deployment tests. The route permissions are listed in `README.md`. ### Auth0 prerequisites diff --git a/rules/registry.toml b/rules/registry.toml index b98635469..e50d5ebd0 100644 --- a/rules/registry.toml +++ b/rules/registry.toml @@ -380,7 +380,7 @@ invocation = "uv run --frozen pytest tests/" [[rule]] id = "sim.deploy.staging-promotes-tested-revision" category = "deployment" -statement = "Every staging deployment passes promote_entrypoint: true, as production deployments do, so the staging service's stable URL serves the entrypoint revision that passed the staging tests." +statement = "Every staging deployment passes promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}, so a run from main moves the staging service's stable URL to the entrypoint revision that passed the staging tests, and a manual run from another branch never changes it." rationale = "projects/policyengine-simulation-entry/DEPLOYMENT.md#promotion-and-rollback" scope = [".github/workflows/*.yml"] tier = "mechanical" From 6369bc43424a98da15a8030c662626ec6d632f8f Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Fri, 30 Oct 2026 20:42:25 +0400 Subject: [PATCH 7/8] Narrow what main-only staging promotion is said to guarantee A manual run from another branch still deploys its Modal gateway and executor to the staging Modal environment and publishes its routing, as every staging deployment does; only the staging service's Cloud Run traffic and its older revisions stay as they were. The deployment guide, rule statement and changelog fragment now say so, say that the check stops an accidental branch run rather than a branch that edits its workflow, and note that the prod environment refuses a branch run. The manual-run paragraph moves after the clean-up, so the stable-URL check again directly follows the traffic change it verifies. The tests share one constant for the required expression. Co-Authored-By: Claude Opus 5.5 --- changelog.d/893.changed.md | 2 +- docs/generated/rules.md | 2 +- .../DEPLOYMENT.md | 35 ++++++++++++------- .../tests/test_deployment_assets.py | 5 ++- .../tests/test_deployment_assets_promotion.py | 9 ++--- .../tests/test_deployment_assets_support.py | 4 +++ rules/registry.toml | 2 +- 7 files changed, 33 insertions(+), 26 deletions(-) diff --git a/changelog.d/893.changed.md b/changelog.d/893.changed.md index a1a625fad..72ccab588 100644 --- a/changelog.d/893.changed.md +++ b/changelog.d/893.changed.md @@ -1 +1 @@ -Promoted each staging deployment's tested Simulation Entrypoint revision to the staging service's stable traffic, as production deployments already do, in both the main deployment's beta job and the Stage 12 comparison path's staging deployment. The staging stable URL now serves the build that passed the staging tests; promotion runs the same stable-URL checks as production's and restores the previous revision when they fail, and the existing clean-up then deletes the staging service's older revisions except the promoted one and the one it replaced. Only a run from main promotes: a manual staging run from another branch tests its candidate but leaves the stable URL and older revisions unchanged. The new rule sim.deploy.staging-promotes-tested-revision requires it. +Promoted each staging deployment's tested Simulation Entrypoint revision to the staging service's stable traffic, as production deployments already do, in both the main deployment's beta job and the Stage 12 comparison path's staging deployment. The staging stable URL now serves the build that passed the staging tests; promotion runs the same stable-URL checks as production's and restores the previous revision when they fail, and the existing clean-up then deletes the staging service's older revisions except the promoted one and the one it replaced. Only a run from main promotes: a manual staging run from another branch tests its candidate but leaves the staging service's Cloud Run traffic and older revisions unchanged. The new rule sim.deploy.staging-promotes-tested-revision requires it. diff --git a/docs/generated/rules.md b/docs/generated/rules.md index f83479118..53cdb5b83 100644 --- a/docs/generated/rules.md +++ b/docs/generated/rules.md @@ -156,7 +156,7 @@ flowchart LR | `sim.deploy.routing-publication-journaled` | Each routing publication is named by its run, attempt and environment and journaled before it is applied; an always-running finalizer resolves it from the journal with the release-gate outcome, a release reports success only after that finalizer completes, and routing jobs never tolerate a failed command. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_release_gates.py`, `projects/policyengine-simulation-entry/tests/test_deployment_assets_routing.py` | `pr.yml:test` | **unproven** | | `sim.deploy.shell-scripts-valid` | Deployment shell scripts have valid shell syntax. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | | `sim.deploy.smoke-tests-strict` | Candidate and stable smoke tests reject a revision mismatch without retrying, and fail if the previous revision does not converge. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets.py` | `pr.yml:test` | **unproven** | -| `sim.deploy.staging-promotes-tested-revision` | Every staging deployment passes promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}, so a run from main moves the staging service's stable URL to the entrypoint revision that passed the staging tests, and a manual run from another branch never changes it. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py` | `pr.yml:test` | yes | +| `sim.deploy.staging-promotes-tested-revision` | Every staging deployment passes promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}, so a run from main moves the staging service's stable Cloud Run traffic to the entrypoint revision that passed the staging tests, and an accidental manual run from another branch neither moves that traffic nor deletes older revisions. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py` | `pr.yml:test` | yes | | `sim.deploy.traffic-changes-exact` | Traffic changes target an exact, validated revision, are reversible, and refuse to run over an intervening promotion. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_traffic.py` | `pr.yml:test` | **unproven** | | `sim.deploy.workflow-shell-safety` | Cloud Run deployment never interpolates GitHub expressions into shell, escapes environment values, and pins secret versions. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_deployment_assets_cloud_run.py` | `pr.yml:test` | **unproven** | diff --git a/projects/policyengine-simulation-entry/DEPLOYMENT.md b/projects/policyengine-simulation-entry/DEPLOYMENT.md index ec2b4beeb..d50a46359 100644 --- a/projects/policyengine-simulation-entry/DEPLOYMENT.md +++ b/projects/policyengine-simulation-entry/DEPLOYMENT.md @@ -35,18 +35,20 @@ require a custom domain. The `prod` environment follows the same sequence automatically, but cannot begin until every `beta` deployment, routing, integration, authentication, promotion, and revision clean-up job has succeeded, so a failed staging -clean-up also stops the production deployment. Both environments' tests use +clean-up also stops the production deployment. A run from a branch other than +`main` skips `beta`'s promotion and clean-up, and the `prod` environment then +refuses the run, since it accepts only `main`. Both environments' tests use the tagged candidate URL, so the currently serving revision remains unchanged during qualification. ## Promotion and rollback Both `beta` and `prod` promote, `beta` only for a run from `main`. After an -environment's complete suite passes, -the workflow verifies that the exact tested revision belongs to that -environment's service and is ready. It also checks that stable traffic still -points to the revision recorded before deployment. If those checks pass, it -assigns all stable service traffic to the tested revision. +environment's complete suite passes, the workflow verifies that the exact +tested revision belongs to that environment's service and is ready. It also +checks that stable traffic still points to the revision recorded before +deployment. If those checks pass, it assigns all stable service traffic to the +tested revision. In `beta`, promotion keeps the staging service's stable URL on the revision that passed the staging tests, so staging callers of that URL, such as the @@ -55,13 +57,6 @@ staging deployment of the **Deploy Stage 12 comparison path** workflow promotes in the same way. Rule `sim.deploy.staging-promotes-tested-revision` requires both. -Both workflows can also be started manually, and the `beta` environment does -not restrict which branch a run may come from. Both therefore pass -`promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}`: a manual run from -a branch other than `main` deploys and tests its candidate revision, but never -moves the staging stable URL to it and never deletes older staging revisions, -so the last revision deployed from `main` stays available for rollback. - The workflow then checks the stable generated Cloud Run URL. It uses the runtime revision header to prove that the stable URL reaches the exact revision that passed candidate tests, and it repeats the deployed authentication checks. @@ -72,6 +67,20 @@ After a successful promotion and deployment, the workflow deletes the service's older revisions. It keeps only the promoted revision and the one it replaced, which `simulation-rollback.yml` can route back to. +Both workflows can also be started manually, and the `beta` environment does +not restrict which branch a run may come from. Both therefore pass +`promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}`. A manual run from +a branch other than `main` deploys and tests its candidate revision, but leaves +the staging service's Cloud Run traffic where it was and deletes no revisions, +so the last revision deployed from `main` stays available for rollback. Such a +run of the main deployment workflow still deploys its Modal gateway and +executor to the staging Modal environment and publishes its routing there, as +every staging deployment does, so callers of the staging stable URL can reach +that branch's Modal services until the next deployment from `main`. The check +reads the branch's own copy of the workflow: it stops an accidental branch run +from promoting, not a branch that edits it. Only a branch restriction on the +`beta` environment would enforce it. + Deployment concurrency prevents two releases from changing the entrypoint service simultaneously. Promotion never targets `LATEST`. diff --git a/projects/policyengine-simulation-entry/tests/test_deployment_assets.py b/projects/policyengine-simulation-entry/tests/test_deployment_assets.py index 503b30295..2b7c4485c 100644 --- a/projects/policyengine-simulation-entry/tests/test_deployment_assets.py +++ b/projects/policyengine-simulation-entry/tests/test_deployment_assets.py @@ -9,6 +9,7 @@ AUTHENTICATED_TESTS, CLOUD_RUN_DEPLOY_SCRIPT, DEPLOY_WORKFLOW, + MAIN_ONLY_PROMOTION, REPOSITORY_ROOT, REUSABLE_DEPLOY_WORKFLOW, SMOKE_SCRIPT, @@ -200,9 +201,7 @@ def _assert_beta_then_prod_release( assert deploy_workflow.count("deploy_existing_stack: true") == 2 # prod always promotes; beta promotes only a run from main. assert deploy_workflow.count("promote_entrypoint: true") == 1 - assert ( - "promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}" in deploy_workflow - ) + assert f"promote_entrypoint: {MAIN_ONLY_PROMOTION}" in deploy_workflow assert "promote_entrypoint: false" not in deploy_workflow assert "entrypoint_environment" not in deploy_workflow assert "entrypoint_environment" not in reusable_workflow diff --git a/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py b/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py index 4295e062a..a49783756 100644 --- a/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py +++ b/projects/policyengine-simulation-entry/tests/test_deployment_assets_promotion.py @@ -1,4 +1,4 @@ -"""Tests for promotion order, Stage 12-only deployments and the sentinel.""" +"""Tests for promotion order, Stage 12-only runs, the sentinel and staging promotion.""" from __future__ import annotations @@ -8,6 +8,7 @@ from test_deployment_assets_support import ( DEPLOY_WORKFLOW, DEPLOYMENT_READY_SCRIPT, + MAIN_ONLY_PROMOTION, REUSABLE_DEPLOY_WORKFLOW, REUSABLE_DEPLOY_WORKFLOW_REFERENCE, STAGE12_DEPLOY_WORKFLOW, @@ -274,12 +275,6 @@ def test_deployed_stack_must_reproduce_the_numerical_sentinel(): ) -# The promotion input every staging deployment passes: true only for a run -# from main, so a manual run from another branch never moves staging's stable -# traffic or deletes its older revisions. -MAIN_ONLY_PROMOTION = "${{ github.ref == 'refs/heads/main' }}" - - def _staging_deployment_job_ids(jobs: dict) -> list[str]: """Return the jobs that run the reusable deployment workflow for staging.""" return [ diff --git a/projects/policyengine-simulation-entry/tests/test_deployment_assets_support.py b/projects/policyengine-simulation-entry/tests/test_deployment_assets_support.py index 0ccb7be42..70ae911b0 100644 --- a/projects/policyengine-simulation-entry/tests/test_deployment_assets_support.py +++ b/projects/policyengine-simulation-entry/tests/test_deployment_assets_support.py @@ -42,6 +42,10 @@ REUSABLE_DEPLOY_WORKFLOW_REFERENCE = ( "./.github/workflows/simulation-deploy.reusable.yml" ) +# The promotion input every staging deployment passes: true only for a run +# from main, so an accidental manual run from another branch never moves +# staging's stable traffic or deletes its older revisions. +MAIN_ONLY_PROMOTION = "${{ github.ref == 'refs/heads/main' }}" STAGING_PROMOTION_FIXTURES = ( REPOSITORY_ROOT / "rules" diff --git a/rules/registry.toml b/rules/registry.toml index e50d5ebd0..1b314011d 100644 --- a/rules/registry.toml +++ b/rules/registry.toml @@ -380,7 +380,7 @@ invocation = "uv run --frozen pytest tests/" [[rule]] id = "sim.deploy.staging-promotes-tested-revision" category = "deployment" -statement = "Every staging deployment passes promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}, so a run from main moves the staging service's stable URL to the entrypoint revision that passed the staging tests, and a manual run from another branch never changes it." +statement = "Every staging deployment passes promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}, so a run from main moves the staging service's stable Cloud Run traffic to the entrypoint revision that passed the staging tests, and an accidental manual run from another branch neither moves that traffic nor deletes older revisions." rationale = "projects/policyengine-simulation-entry/DEPLOYMENT.md#promotion-and-rollback" scope = [".github/workflows/*.yml"] tier = "mechanical" From 261957b335cdce576ad44ec9aa41bfac27eeec32 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Fri, 30 Oct 2026 20:50:05 +0400 Subject: [PATCH 8/8] Say which Modal services a branch staging run still deploys The deployment guide named only the gateway and executor and said they last until the next deployment from main. A branch run of the main deployment workflow also deploys the Stage 12 workers, the Stage 12 workflow deploys only those, and routes and versions that main doesn't deploy can stay after main's next deployment. It also says that a branch restriction on the beta environment would refuse every branch run's candidate deployment and tests. Co-Authored-By: Claude Opus 5.5 --- .../DEPLOYMENT.md | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/projects/policyengine-simulation-entry/DEPLOYMENT.md b/projects/policyengine-simulation-entry/DEPLOYMENT.md index d50a46359..76038ce31 100644 --- a/projects/policyengine-simulation-entry/DEPLOYMENT.md +++ b/projects/policyengine-simulation-entry/DEPLOYMENT.md @@ -72,14 +72,21 @@ not restrict which branch a run may come from. Both therefore pass `promote_entrypoint: ${{ github.ref == 'refs/heads/main' }}`. A manual run from a branch other than `main` deploys and tests its candidate revision, but leaves the staging service's Cloud Run traffic where it was and deletes no revisions, -so the last revision deployed from `main` stays available for rollback. Such a -run of the main deployment workflow still deploys its Modal gateway and -executor to the staging Modal environment and publishes its routing there, as -every staging deployment does, so callers of the staging stable URL can reach -that branch's Modal services until the next deployment from `main`. The check -reads the branch's own copy of the workflow: it stops an accidental branch run -from promoting, not a branch that edits it. Only a branch restriction on the -`beta` environment would enforce it. +so the last revision deployed from `main` stays available for rollback. + +Promotion covers only the Cloud Run entrypoint. A branch run still deploys the +branch's Modal services to the staging Modal environment and publishes their +routing there, so callers of the staging stable URL can reach them: the main +deployment workflow deploys the gateway, the executor and the Stage 12 workers, +and the Stage 12 workflow deploys the Stage 12 workers. A later deployment from +`main` replaces the apps it deploys, but routes and versions that it doesn't +deploy can stay. + +The check reads the branch's own copy of the workflow: it stops an accidental +branch run from promoting, not a branch that edits it. Only a branch +restriction on the `beta` environment would enforce it, and that would also +refuse every branch run's candidate deployment and tests, since each of the +reusable workflow's deployment jobs runs in that environment. Deployment concurrency prevents two releases from changing the entrypoint service simultaneously. Promotion never targets `LATEST`. @@ -115,8 +122,7 @@ value fails the deployment before Cloud Run changes. A new value takes effect at the next entrypoint deployment, which `prod`, and `beta` for a run from `main`, promote to stable traffic once their tests pass. A `beta` value therefore applies to every caller of the staging stable URL as well as to the -deployment tests. The route -permissions are listed in `README.md`. +deployment tests. The route permissions are listed in `README.md`. ### Auth0 prerequisites