Add controlled AWS validation evidence gates

backend/tests/test_api.py

@@ -17,6 +17,8 @@
 os.environ["DATABASE_URL"] = f"sqlite+pysqlite:///{test_db}"
 os.environ["CORS_ORIGINS"] = "http://testserver"
 os.environ["DEMO_MODE"] = "false"
+os.environ["PROMETHEUS_NAMESPACE"] = "test"
+os.environ["PROMETHEUS_SERVICE"] = "backend"
 
 from app import main as main_module  # noqa: E402
 

docs/aws-addons.md

@@ -1,6 +1,10 @@
 # AWS EKS Add-Ons
 
-This document prepares the Kubernetes add-ons used after Terraform creates the short-lived AWS demo cluster. AWS Load Balancer Controller, Metrics Server, and CloudWatch Observability are part of the main demo path; kube-prometheus-stack is optional for a deeper Grafana run.
+This document prepares the Kubernetes add-ons used after Terraform creates the
+short-lived AWS demo cluster. AWS Load Balancer Controller, Metrics Server, and
+CloudWatch Observability are part of every AWS path. kube-prometheus-stack is
+optional for an app-only demo and required for the controlled v1.2.1
+request-SLI validation.
 
 Do not run these commands until:
 
@@ -20,7 +24,7 @@ The AWS Load Balancer Controller instructions are pinned to the versions referen
 Other add-on versions:
 
 - Metrics Server Helm chart: `3.13.0`
-- Optional kube-prometheus-stack Helm chart: `86.1.0`
+- kube-prometheus-stack Helm chart: `86.1.0` when running request-SLI validation
 - Amazon CloudWatch Observability EKS add-on: use the default version selected by EKS for the cluster version
 
 ## 1. Connect To EKS
@@ -276,9 +280,14 @@ Capture screenshots of:
 - `kubectl get pods,svc,ingress,hpa -n cloudops -o wide`
 - Live CloudOps dashboard on the ALB URL
 
-Optional if kube-prometheus-stack is installed:
+Required for the controlled v1.2.1 validation:
 
 - `kubectl get pods -n monitoring`
+- Healthy scoped Prometheus targets and recording rules
+- Compiled request-SLI evidence
+
+Optional:
+
 - Grafana dashboard showing pod CPU/memory
 - Grafana HPA current and desired replica panels
 

docs/aws-demo-checklist.md

@@ -95,7 +95,7 @@ Install:
 - AWS Load Balancer Controller
 - Metrics Server
 - CloudWatch Observability add-on
-- Optional kube-prometheus-stack for Grafana dashboards
+- kube-prometheus-stack for the required v1.2.1 request-SLI evidence
 
 ## Deploy App
 
@@ -119,8 +119,12 @@ Verify:
 kubectl get pods,svc,ingress,hpa -n cloudops -o wide
 kubectl rollout status deployment/cloudops-cloudops-sre-platform-backend -n cloudops
 kubectl rollout status deployment/cloudops-cloudops-sre-platform-frontend -n cloudops
+helm test cloudops -n cloudops --logs --timeout 5m
 ```
 
+Retain the sanitized Helm test pass/fail result in the runtime-control record.
+Run the test while the chart NetworkPolicies remain enabled.
+
 ## Capture Demo Artifacts
 
 Use:
@@ -130,6 +134,7 @@ docs/demo-validation-checklist.md
 docs/hpa-demo.md
 docs/observability.md
 docs/rds-connectivity-secret-rotation-runbook.md
+docs/evidence/aws-validation-records.md
 ```
 
 Minimum evidence set:
@@ -146,15 +151,39 @@ Minimum evidence set:
 - CloudWatch backend/frontend logs
 - Terraform apply output
 - Terraform destroy confirmation
+- Sanitized runtime-control, Terraform lifecycle, zero-resource closure, and
+  findings records based on `docs/evidence/templates/`
 
-Optional:
+Required for the controlled v1.2.1 validation:
 
-- Grafana CPU/HPA graphs if kube-prometheus-stack is installed
 - Sanitized Prometheus SLI evidence generated by the read-only traffic profile
   and collector in `docs/grafana-demo.md`
+- Healthy scoped Prometheus targets and recording rules
+
+Optional:
+
+- Grafana CPU/HPA screenshots; these are presentation aids, not substitutes
+  for machine-readable evidence
 
 ## Destroy Same Day
 
+Capture scoped identifiers privately before uninstalling the ingress and before
+Terraform removes its outputs:
+
+```bash
+PROJECT_NAME="cloudops-sre-platform"
+ENVIRONMENT="demo"
+AWS_REGION="$(terraform -chdir=infra output -raw aws_region)"
+CLUSTER_NAME="$(terraform -chdir=infra output -raw cluster_name)"
+ALB_HOSTNAME="$(
+  kubectl get ingress cloudops-cloudops-sre-platform \
+    -n cloudops \
+    -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
+)"
+ALB_NAME="${ALB_HOSTNAME%%.*}"
+test -n "$ALB_NAME"
+```
+
 Uninstall app/add-ons:
 
 ```bash
@@ -179,9 +208,34 @@ terraform -chdir=infra destroy
 Verify expensive resources are gone:
 
 ```bash
-aws eks list-clusters
-aws elbv2 describe-load-balancers
-aws rds describe-db-instances
-aws ec2 describe-nat-gateways
-aws ec2 describe-instances
+aws eks list-clusters --region "$AWS_REGION" \
+  --query "length(clusters[?@ == '$CLUSTER_NAME'])"
+aws rds describe-db-instances --region "$AWS_REGION" \
+  --query "length(DBInstances[?DBInstanceIdentifier == '$CLUSTER_NAME-postgres'])"
+aws ec2 describe-nat-gateways --region "$AWS_REGION" \
+  --filter "Name=tag:Project,Values=$PROJECT_NAME" \
+  --query "length(NatGateways[?State != 'deleted'])"
+aws elbv2 describe-load-balancers --region "$AWS_REGION" \
+  --query "length(LoadBalancers[?LoadBalancerName == '$ALB_NAME'])"
+aws ec2 describe-instances --region "$AWS_REGION" \
+  --filters "Name=tag:Project,Values=$PROJECT_NAME" \
+    "Name=instance-state-name,Values=pending,running,stopping,stopped" \
+  --query "length(Reservations[].Instances[])"
+aws ecr describe-repositories --region "$AWS_REGION" \
+  --query "length(repositories[?starts_with(repositoryName, '$PROJECT_NAME/')])"
+aws secretsmanager list-secrets --region "$AWS_REGION" \
+  --query "length(SecretList[?starts_with(Name, '$CLUSTER_NAME/')])"
+aws logs describe-log-groups --region "$AWS_REGION" \
+  --query "length(logGroups[
+    ?contains(logGroupName, '$CLUSTER_NAME')
+    || starts_with(logGroupName, '/$PROJECT_NAME/$ENVIRONMENT/')
+  ])"
+aws resourcegroupstaggingapi get-resources --region "$AWS_REGION" \
+  --tag-filters "Key=Project,Values=$PROJECT_NAME" \
+  --query "length(ResourceTagMappingList)"
 ```
+
+Every command must return integer `0`. Also record
+`terraform -chdir=infra state list | wc -l` as zero. A failed AWS query is
+`inconclusive`, never zero; use the closure template in
+`docs/evidence/templates/zero-resource-closure.example.json`.

docs/aws-validation-v1.2.1-spec.md

@@ -0,0 +1,147 @@
+# AWS Validation v1.2.1 Specification
+
+## Purpose
+
+Run one short-lived, controlled AWS validation of the v1.2.0 application and
+the current reliability controls. The objective is to produce explainable
+evidence, not to tune the workload until it reports perfect results.
+
+## Fixed Point
+
+- Application release: `v1.2.0`
+- Git commit: `c38d2db36ec4438cc3ec0c629abe36c8bb0e04c8`
+- AWS region: `us-east-1`
+- EKS version: `1.35` under standard support at preflight
+- Terraform state: local, with zero managed resources at preflight
+- AWS resource baseline: zero resources tagged
+  `Project=cloudops-sre-platform` at preflight
+
+## Cost And Time Boundary
+
+- Existing AWS cost budget: `My Zero-Spend Budget`
+- Budget limit: USD 1, with an actual-cost notification above USD 0.01
+- Initiate teardown no later than `2026-07-26T00:30:00Z`
+- Complete teardown and zero-resource checks no later than
+  `2026-07-26T01:30:00Z`
+- Initiate teardown within 150 minutes of Terraform apply even if the absolute
+  deadline is later.
+- Do not leave the environment running overnight.
+
+The budget is an alert, not an enforcement control. The teardown deadline is
+the primary cost control.
+
+The deadline was revised before provisioning at `2026-07-25T21:50:00Z` after
+the pre-apply evidence review took longer than planned. The extension preserves
+same-day local teardown and does not relax the USD 1 budget alert.
+
+## Public Evidence Seams
+
+### HPA k6 summary
+
+The HPA workload must emit a machine-readable JSON summary containing:
+
+- schema, profile, and per-execution run identifiers
+- completion timestamp and k6 version
+- configured stages, target path, and CPU duration
+- request count, response-check passes/failures, dropped iterations, and
+  observed duration
+- HTTP status classes, unexpected statuses, transport errors, and invalid
+  functional responses
+- p50, p90, p95, and p99 latency
+- one sanitized failure event for every categorized failed request
+
+### Kubernetes timeline
+
+The run must retain timestamped observations covering:
+
+- HPA current/desired replicas and current/target CPU
+- backend Deployment ready/available replicas
+- backend pod readiness and restart totals
+- baseline, maximum scale-out, and stable scale-in
+- the same per-execution run identifier used by k6 and the run context
+
+### Compiled evidence
+
+The evidence compiler must reject:
+
+- the wrong profile or schema
+- missing or internally inconsistent request/check/status counts
+- an absent baseline or scale-out observation
+- no return to the baseline replica count
+- a missing CPU-threshold breach or scale-out outside the k6 run window
+- pod unavailability or restarts that are omitted from the verdict
+- missing or mismatched run identity, fixed commit, application version,
+  workload hash, image digests, k6 identity, or cluster version
+- missing retained failure events, including invalid `200` response bodies
+- an existing output file
+
+The compiled record must retain workload and source-artifact SHA-256 hashes and
+state limitations explicitly.
+
+## AWS Validation Gates
+
+### Platform
+
+- Terraform plan contains only the reviewed CloudOps resources.
+- EKS nodes become `Ready`.
+- AWS Load Balancer Controller, Metrics Server, and
+  kube-prometheus-stack become healthy.
+- Backend and frontend Deployments complete their rollouts.
+- Helm test traffic succeeds while NetworkPolicies are enabled.
+
+### Access And Runtime Security
+
+- ALB ingress is restricted to the operator `/32`.
+- AWS presentation mode reports writes disabled.
+- A public mutation request returns `403`.
+- Public `/api/metrics` returns `404`.
+- Backend `/live` and `/ready` return `200` while PostgreSQL is reachable.
+- Application containers run as non-root with read-only root filesystems,
+  dropped capabilities, and no privilege escalation.
+- Backend and frontend PodDisruptionBudgets are present.
+
+### HPA Workload
+
+- Begin with two backend replicas.
+- Produce a recorded CPU-driven scale-out above the baseline.
+- Return to two replicas and remain there for three consecutive observations.
+- Record every non-200 response and transport error.
+- Treat a non-zero failure rate as a finding to explain, not a value to hide.
+- The existing threshold remains below 5%; the evidence report also calculates
+  whether the run stayed below 1%.
+
+### Request SLI Workload
+
+- Run the read-only traffic profile separately after HPA stabilization.
+- Run for at least seven minutes with zero dropped iterations.
+- Record check failures rather than filtering them out.
+- Require healthy scoped Prometheus targets and recording rules.
+- Capture eligible request rate, server-error availability, and
+  requests-at-or-below-500-ms ratio.
+
+## Evidence Package
+
+Commit only sanitized artifacts:
+
+- HPA k6 summary
+- Kubernetes HPA timeline
+- compiled HPA evidence
+- SLI k6 summary and compiled Prometheus evidence
+- runtime-control verification
+- Terraform apply/destroy summaries
+- zero-resource closure record
+- findings report with passed, failed, and inconclusive controls
+
+Do not commit account IDs, credentials, secret values, database endpoints,
+operator IPs, ALB hostnames, or kubeconfig contents.
+Keep the raw k6 JSONL private because it may include runtime lines outside the
+whitelisted failure-event schema. The compiled HPA record carries sanitized
+events and the raw log's SHA-256.
+
+## Claim Boundary
+
+This run may validate the declared EKS deployment, HPA behavior, NetworkPolicy
+path, runtime security settings, request-SLI pipeline, and same-day teardown.
+It does not establish production capacity, monthly SLO compliance, RDS
+failover, backup/restore behavior, multi-region resilience, authentication, or
+HTTPS.

docs/evidence/aws-validation-records.md

@@ -0,0 +1,79 @@
+# AWS Validation Records
+
+Use one `run_id` across every v1.2.1 artifact. Raw command output stays in the
+private run directory. Only the normalized records described here may be
+copied into the repository after a secret and identifier review.
+
+Copy the examples in `docs/evidence/templates/` into the private run directory
+before provisioning. Replace every `pending` status during the run. A final
+record may use only `passed`, `failed`, or `inconclusive`; never convert missing
+evidence into a pass.
+
+## Runtime Controls
+
+Record the observed status or value for each required control:
+
+- public mutation returns `403`
+- public `/api/metrics` returns `404`
+- backend `/live` and `/ready` return `200`
+- presentation mode reports writes disabled
+- containers are non-root, read-only, without privilege escalation, and with
+  all capabilities dropped
+- backend and frontend PodDisruptionBudgets exist
+- ALB source range matches the private operator `/32`
+
+The public record states only whether the configured range matched the private
+expected value. It must not contain the IP, ALB hostname, database endpoint, or
+Kubernetes Secret data.
+
+## Terraform Lifecycle
+
+Keep the binary plan, `terraform show -json`, apply output, state, and destroy
+output private. Record only timestamps, aggregate action counts, and aggregate
+state-resource counts in the sanitized lifecycle record.
+
+Before apply, derive plan action counts from the private plan JSON:
+
+```bash
+terraform -chdir=infra show -json "$PRIVATE_PLAN" |
+  jq '
+    [.resource_changes[].change.actions | join(",")]
+    | group_by(.)
+    | map({(.[0]): length})
+    | add
+  '
+```
+
+After apply and after destroy, record the count returned by:
+
+```bash
+terraform -chdir=infra state list | wc -l
+```
+
+The final state count must be zero. A successful Terraform command alone is not
+closure evidence.
+
+## Zero-Resource Closure
+
+After destroy, record zero counts for the Terraform state and the scoped AWS
+checks in the closure template. Query EKS clusters, RDS instances, NAT
+gateways, load balancers, EC2 instances, ECR repositories, Secrets Manager
+secrets, project log groups, and project-tagged resources. Keep raw AWS output
+private; the public record contains the command family, region, checked time,
+and integer count only.
+
+If a query fails, mark that check `inconclusive`. Do not record zero.
+
+## Findings
+
+The findings report is the control ledger for the run. Include every platform,
+access/runtime, HPA, SLI, and teardown gate from
+`docs/aws-validation-v1.2.1-spec.md`. Each control has one status:
+
+- `passed`: the expected condition is supported by a retained artifact
+- `failed`: the observed condition contradicted the expectation
+- `inconclusive`: the evidence was absent, incomplete, or the check failed
+
+Link findings only to repository-relative sanitized artifact paths and their
+SHA-256 values. The summary counts must equal the control entries. Keep failed
+and inconclusive controls in the report.