feat: harden AWS integration security and deployment evidence

[CoreyLeath-code]

Summary

Completes the next production-hardening roadmap tranche on top of the CloudWatch observability branch while preserving the repository's evidence-first/no-overclaim rule.

This is a stacked PR based on feat/aws-cloudwatch-observability.

What changed

Real development Amazon MSK integration

• Add an optional private IAM-authenticated MSK Serverless development cluster in Terraform.
• Reuse either an existing msk_cluster_arn or the optional development cluster as the Lambda MSK event source.
• Add an opt-in SNS -> SQS probe sink for end-to-end measurement.
• Add a manual OIDC-authenticated workflow designed for a VPC-connected self-hosted runner.
• Add scripts/aws_msk_e2e.py to send synthetic trip.completed probes through MSK -> Lambda -> SQS -> Lambda -> SNS -> probe SQS and record min/mean/p50/p95/p99/max latency.
• Record cost separately as an explicit request-count/runtime estimate with supplied pricing inputs; it is not mislabeled as AWS billing data.
• Check in evidence/aws-msk-integration-results.json with status: not_run until a real authorized AWS run occurs.

Authentication, Secrets Manager, and PII boundaries

• Add opt-in RS256 OIDC/Cognito-style JWT validation for /drivers, /drivers/{driver_id}, and /count.
• Keep /health and /ready unauthenticated for infrastructure probes.
• Add runtime AWS Secrets Manager resolution for the Redis URL with a local environment fallback.
• Add optional KMS key, Secrets Manager metadata, and least-privilege reader-policy Terraform resources without storing secret values in Terraform state.
• Reject direct PII fields in both Lambda stages before SQS/SNS fan-out.
• Stop the in-memory event bus and driver consumer from logging raw event bodies/coordinates/input validation details.

Kubernetes environments and promotion/rollback evidence

• Replace the hard-coded yourdockerhub/api-gateway:latest deployment image.
• Add Kustomize base plus dev/staging/prod overlays.
• Add rolling-update safety, revision history, probes, and resource requests/limits.
• Add render-only or real-apply promotion workflow with manifest SHA-256 and revision evidence.
• Add explicit rollback workflow with before/after revision evidence.
• Require immutable @sha256: images for production promotion before any apply.
• Block staging/prod applies while OIDC issuer/audience placeholders remain unresolved.

Coverage and validation

• Add route-level API auth wiring tests.
• Add consumer privacy/logging tests.
• Add broker pre-connect contract tests while preserving Docker round-trip integration tests.
• Make root Kafka/Redis/RabbitMQ adapters directly importable instead of requiring the integration test's fake-package workaround.
• Add Lambda PII-boundary tests.
• Add pure unit tests for MSK evidence statistics/cost calculation and Kubernetes release evidence.
• Add Kubernetes overlay validation workflow.
• Expand AWS validation workflow while keeping regular tests credential-free.

Documentation

• Rewrite README around implemented vs measured vs target vs not-run evidence.
• Add docs/aws-msk-integration.md.
• Add docs/security-and-pii.md.
• Add docs/kubernetes-promotion.md.
• Update AWS serverless architecture docs, .env.example, and changelog.
• Remove these completed items from the future roadmap.

Evidence / production boundary

No real AWS MSK latency/cost result or Kubernetes deployment/rollback result is claimed by this PR. The code, Terraform, workflows, and evidence schemas are implemented, while checked-in cloud/deployment evidence remains not_run until an authorized environment actually executes those workflows.

Remaining production gaps are documented explicitly: durable idempotency and DLQ replay, fine-grained authorization, distributed tracing, measured SLO/error-budget policy, secret rotation operations, capacity/failover/DR evidence, payment/trip consistency controls, and production privacy/audit policy.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 22fca0ae-612a-480e-b65b-35a233d97dd6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 584962572f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve the existing Deployment selector during promotion

When this manifest is applied over a Deployment created from the previous version, kubectl apply rejects the update because spec.selector is immutable and this changes it from app: api-gateway to app.kubernetes.io/name: api-gateway. The new promotion workflow applies the manifest to the same api-gateway object, so the first promotion to any existing environment cannot proceed; retain the original selector label or explicitly implement a safe replacement migration.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Permit the measurement runner to reach the MSK security group

For the documented self-hosted VPC runner, this self-referencing rule only admits sources already attached to the newly created MSK security group. The workflow supplies only the VPC and subnet IDs and never attaches this generated group to the runner or adds an ingress rule for the runner's security group, so scripts/aws_msk_e2e.py cannot connect to port 9098 even when the runner has a route into the VPC. Accept a runner security-group input or otherwise authorize the producer explicitly.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Catch JWKS lookup failures as authentication errors

When an unauthenticated caller supplies a malformed JWT or an unknown kid, get_signing_key_from_jwt() can raise before entering the try block, so FastAPI returns an unhandled 500 rather than the intended 401 response. Move the signing-key lookup into the guarded verification path and translate its PyJWT/JWK errors to the same authentication failure response.

Useful? React with 👍 / 👎.