Follow-up: Fix review-app setup and restore deployed behavior gate

[justin808]

Environment

• Affected surface: repository review-app workflow
• Source PR head: c524f0ac736287ef902f32b12e5f82ccdb36b1cc

Steps To Reproduce

1. Run the configured review-app deployment workflow for PR Validate the TanStack starter against React on Rails 17 RC12 #193 at the source head.
2. Observe that initial setup exits before image build, deployment, endpoint creation, or behavioral smoke begins.

Expected Behavior

The workflow should establish a usable review app and run deployed behavioral verification against the exact PR head.

Actual Behavior

The explicit attempt stopped during initial setup. Image build, deployment, endpoint creation, and deployed smoke did not run. The RC12 package update itself passed local validation, hosted CI, hosted review, and all review threads; no React on Rails RC12 code regression was found.

Acceptance criteria

• Diagnose and repair the repository-owned review-app setup/configuration failure.
• Re-run the workflow on the repaired current head and retain the exact tested commit plus public workflow evidence.
• Confirm image build, deployment, and a usable review endpoint complete successfully.
• Run behavioral smoke that verifies expected server-rendered HTML, the headline RSC/client interaction, and no new browser console errors.
• Clean up the temporary review app after evidence is retained.

Evidence

• RC12 validation PR: Validate the TanStack starter against React on Rails 17 RC12 #193
• Setup-stage workflow attempt: https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29553512609
• Fleet tracker: Release gate: react_on_rails 17.0.0 react_on_rails#3823

This follow-up owns the review-app infrastructure gate only; it does not identify an RC12 runtime defect.

[justin808]

qa-evidence v1 — post-merge replay: FAIL (external gate)

• tested_at: 2026-07-18T08:30:36Z
• Exact merged default: 2d409e8df604b76544fe3d47bb338825bbb86f28 (contains Use stable configured-template review-app validation #195; remote main remained exactly reachable after cleanup)
• Verification commit: 1180dac29ec3dfc07399048c7f72556801f2b2fc (one inert marker only)
• Disposable draft PR: QA: Verify merged review-app gate (DO NOT MERGE) #196 — closed unmerged; remote and local verification branches deleted
• Default-owned PR validation: PASS — https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29637204905
• Trusted comment deployment: FAIL — https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29637210233
  • source/config resolution: PASS
  • review-app setup: PASS
  • image/build: PASS
  • deploy: FAIL during the release database-preparation path
  • endpoint reachability: UNKNOWN (not reached)
  • SSR/RSC behavior: UNKNOWN (not reached)
  • client headline/interaction/navigation: UNKNOWN (not reached)
  • browser console errors: UNKNOWN (not reached)
  • cheap unhappy path: UNKNOWN (not safe/meaningful without a healthy endpoint)
• Sanitized cause signature: ActiveRecord::DatabaseConnectionError / PG::ConnectionBad with database password-authentication rejection during release preparation. Classification: repo-owned review-app template/config mapping manifested at runtime, not workflow dispatch/build and not an external-platform outage. The review template consumes a database-password reference that differs from the shared PostgreSQL/persistent-app reference in the repo-owned templates. No raw deployment logs or private endpoint/config identifiers are included here.
• Explicit cleanup: PASS — https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29637397219
• PR-close cleanup: PASS — https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29637538951
• Cleanup verification: PASS — both deletion paths completed their delete and finalize steps; PR is closed with mergedAt: null; verification refs are absent; exact default remains reachable.

Verdict: external-gate-failing. No waiver. Keeping #194 open because deploy, endpoint, SSR/RSC, client, console, and unhappy-path requirements did not all pass.

[justin808]

QA Evidence

• QA lane: ror17-fleet-c-wave1-tanstack-reviewapp-1 on the disposable diagnostic branch; claim active; heartbeat in progress
• Scope checked: exact fresh-default review-app composition, deployment metadata, provider relationship categories, rollout readiness, cleanup, and default reachability
• Tested at: draft PR QA: replay fresh-default review-app deployment #198 at head 1f9432d
• Automated checks: deploy run https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29645170714 failed; explicit cleanup run https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29645398902 passed; close cleanup run https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29645432640 passed
• Manual checks: not run because deployment did not reach a healthy endpoint; endpoint, server-rendered output, SSR, configured RSC behavior, client navigation, console, and unhappy-path checks are UNKNOWN
• Findings: blocked; the repository review database reference does not match the configured shared grant target, its referenced provider object is absent, while the configured shared object and review grant subject are present and shared resource routing aligns
• QA required: yes
• QA required rationale: the failure occurs only in the hosted review-app lifecycle and required an exact fresh-default replay
• QA lane status: blocked
• Release-blocking status: blocked
• Process-gap disposition: schema

Recommended disposition: repo fix justified. Keep issue #194 open. PR #197 should remain unmerged because its canonical-copy mapping does not represent the shared disposable composition. The minimal follow-up is a review-template reference correction to the configured shared grant target plus a repository validation that enforces that relationship.

[justin808]

Fresh-default verification replay — external gate failing

• Default tested: 97733defc43f41e63d81df97f81683972a32843f.
• Empty verification commit: 92fb33dd6f8cdfa8471e555bf14d9216c807d227 on temporary PR #200, now closed unmerged.
• Fresh setup and repository-owned pre-push validation passed locally, including quality, security, static checks, 119 RSpec examples, asset compilation, and the two-test Playwright root smoke.
• The PR-open guard run succeeded while all create/build/deploy/readiness steps were skipped, confirming that PR creation alone did not deploy.
• Exactly one deployment request was made. The deployment run completed with the fresh review-app setup and image build passing, then failed at the deploy/rollout stage. Health readiness and endpoint retrieval were skipped. Root cause: UNKNOWN; protected logs and deployment details were intentionally excluded from this public evidence.
• Browser acceptance is UNKNOWN because no ready endpoint was produced: usable endpoint, SSR/RSC HTML evidence, hydration, RSC/client navigation, real interaction, normal-path console/page errors, and the not-found path were not run. Build success was not treated as a waiver.
• Exactly one explicit teardown request was made. The teardown run completed successfully.
• The PR close-trigger cleanup run also completed successfully. The temporary remote branch was deleted.

Issue state: keep open. Next action: diagnose the external rollout failure through the protected operator path, then launch a new fresh-default verification lane rather than retrying this one.

[justin808]

Protected retrospective diagnosis (public-safe)

• Existing verification: https://github.com/shakacode/react-on-rails-starter-tanstack/actions/runs/29677604599
• Classification: the deployment platform rejected rollout at an external authorization/access-control boundary.
• Repo-owned remediation: none identified. The verified default and workflow are unchanged since the failed replay.
• Next gate: an operator must restore deployment authorization and verify it independently.
• Replay: justified only after that authorization is proven restored.

Issue remains open as external-gate-failing. No application location, log content, credential detail, or protected deployment identifier is included.

[justin808]

Root cause confirmed — stale placeholder credential, not an external gate

Reproduced end to end on a disposable PR (#205, closed unmerged; review app and policy binding torn down and verified clean).

Root cause

The password field of Control Plane secret react-on-rails-starter-tanstack-staging-pg still contains the committed placeholder value from .controlplane/templates/postgres.yml — it was never replaced with the real credential. .controlplane/readme.md even warns to "Replace the placeholder password before serious staging testing."

Review apps resolve that secret through {{SHARED_SECRET_DATABASE}} in .controlplane/templates/app-review.yml, so bin/rails db:prepare in .controlplane/release_script.sh authenticates with the placeholder, Postgres rejects it, and the deploy exits 64.

The persistent staging app does not hit this, because its deployed GVC was hand-patched at some point to carry the real database password as an inline literal instead of the cpln://secret/... reference. So staging kept working while every review app failed — which is why this looked like an intermittent or external problem.

Reproduction

Run 29974807438 — review-app setup PASS, image build PASS, deploy FAIL:

ActiveRecord::DatabaseConnectionError: There is an issue connecting to your database
  with your username/password, username: react_on_rails_starter_tanstack.
PG::ConnectionBad: ... FATAL:  password authentication failed for user "react_on_rails_starter_tanstack"
[2026-07-23:02:40:13]: Failed to run DB migrations
ERROR: Failed to run release script.
Process completed with exit code 64.

Why this was missed for so long

Three traps, all worth recording:

1. The 2026-07-19 retrospective classified this as an external authorization/access-control boundary with "no repo-owned remediation identified." Incorrect. Nothing external rejected the rollout — Postgres rejected a placeholder password.

2. Verifying the credential from inside the Postgres container gives a false PASS. pg_hba.conf on the shared server is:

   host  all  all  127.0.0.1/32   trust           <- loopback: no password check at all
   host  all  all  all            scram-sha-256   <- what review apps actually hit
   

   Connecting over 127.0.0.1 succeeds with any password — confirmed with a deliberately-wrong-password control, which also connected. Only a test over the non-loopback address exercises scram-sha-256.

3. The shared secret looks correctly wired. The grant, policy, placeholder substitution, and binding are all correct. Only the stored value is wrong, and no amount of config inspection reveals that.

Ruled out — verified against live infrastructure, please do not re-chase

Hypothesis	Verdict
Shared-grant mapping drift in app-review.yml	Mapping is correct; {{SHARED_SECRET_DATABASE}} resolves to the declared grant
Referenced provider object absent (per 2026-07-18)	Both secret and policy exist
bindings: [] means the grant is never applied	Applied correctly and held continuously across the failure. Full lifecycle, polled every 10s: EMPTY at 02:35:20Z → reveal -> react-on-rails-starter-tanstack-review-pr-205-identity at 02:36:27Z → EMPTY at 02:44:42Z on teardown. The release script failed at 02:40:13Z, squarely inside that window. cpflow binds and unbinds exactly as documented
DB role missing / wrong hardcoded username	Role react_on_rails_starter_tanstack exists, has a password, has CREATEDB
Firewall blocks cross-GVC traffic	inboundAllowType: same-org permits review-app GVCs
Workflow passes --skip-secrets-setup	No skip flags anywhere in this repo
React on Rails RC12 / 17.0.0 regression	Unrelated; failure is at DB auth, before app boot

Fix — update the secret, do NOT alter the role

Set the password field of react-on-rails-starter-tanstack-staging-pg to the database password the staging GVC currently carries inline, preserving the existing username field.

Warning

Do not "fix" this with ALTER ROLE react_on_rails_starter_tanstack PASSWORD .... The role's current password is the one the persistent staging app depends on. Changing the role to match the placeholder-bearing secret would break staging. The secret is the wrong side of the mismatch — correct the secret.

No change is required in this repository. Templates, controlplane.yml, workflows, and config/database.yml are all correct.

Follow-ups worth separate issues

• Staging drift: the deployed staging GVC carries the database password as an inline literal rather than the cpln://secret/... reference its template declares, so staging and review apps resolve credentials by different mechanisms. Reconciling staging onto the shared secret would make this class of drift impossible.
• PR Fix disposable review app shared grant mapping #199 accuracy: its description says it shipped "one scalar template correction plus one focused regression spec," but merge commit 97733de contains only spec/controlplane/template_mapping_spec.rb. .controlplane/templates/app-review.yml has been unchanged since [codex] Move review apps to shared Postgres #151. That spec compares controlplane.yml against the template only, so it passes while deploys fail — it cannot detect a wrong stored value.