CloudOps SRE Platform

CloudOps SRE Platform is a reliability-operations dashboard for Amazon EKS. It tracks service health, deployments, incidents, MTTR, request SLIs, and catalog-derived operational indicators.

The project is built with React, FastAPI, PostgreSQL, Docker, Helm, Terraform, GitHub Actions, Amazon EKS, Amazon RDS, ALB Ingress, Secrets Manager, CloudWatch, HPA, and k6.

System Context

Reliability investigations require service health, recent deployments, active incidents, ownership, recovery duration, and SLO status. CloudOps SRE Platform models those relationships in one application and couples them with deployment, observability, and operational controls for Amazon EKS.

What The Platform Does

• Tracks services with owner, environment, health status, SLO target, service URL, and current version
• Records deployments with service version, commit SHA, deployment status, and deployment time
• Manages incidents with P1-P4 severity, investigation state, timeline updates, resolution, and MTTR
• Shows reliability dashboard metrics for open incidents, failed deployments, average MTTR, service health, SLOs, and error budgets
• Exposes Prometheus request counters and latency histograms using bounded route-template labels
• Provides an optional ServiceMonitor and Prometheus recording rules for request-based server-error availability and latency SLIs
• Reads scoped Prometheus recording rules through /sli and displays explicit source state
• Evaluates 99.9% availability error-budget burn with fast and slow multi-window alerts
• Supports a read-only operations mode that blocks mutation endpoints and hides write controls
• Includes a read-only traffic profile, k6 run record, and collector for Prometheus target, rule, and SLI snapshots
• Emits structured JSON backend logs with request correlation IDs for troubleshooting
• Includes frontend loading, retry, and API failure states for operational views
• Provides a bounded CPU endpoint for HPA validation under controlled k6 load

Catalog-derived SLO and error-budget values are operational proxies. The Metrics view separates them from request availability, latency, and eligible traffic queried from scoped Prometheus recording rules.

Architecture

flowchart LR
    user["Cloud / SRE Operator"] --> alb["Application Load Balancer"]
    alb --> ingress["Kubernetes Ingress"]
    ingress --> frontend["React + Nginx frontend pods"]
    frontend --> backend["FastAPI backend pods"]
    backend --> rds["Amazon RDS PostgreSQL"]
    secrets["AWS Secrets Manager"] --> k8ssecret["Kubernetes Secret"]
    k8ssecret --> backend

    gha["GitHub Actions"] --> ecr["Amazon ECR"]
    gha --> deployRunner["VPC-connected deploy runner"]
    deployRunner --> eks["Amazon EKS"]
    ecr --> eks
    terraform["Terraform"] --> aws["VPC / EKS / ECR / RDS / IAM / Secrets / CloudWatch"]
    metrics["Metrics Server"] --> hpa["Backend HPA"]
    hpa --> backend
    backend --> prometheusEndpoint["Cluster-local /metrics"]
    serviceMonitor["Optional ServiceMonitor"] -->|"selects backend Service"| backend
    serviceMonitor --> operator["Prometheus Operator"]
    prometheusRule["Optional PrometheusRule"] --> operator
    operator -->|"configures"| prometheus["Optional Prometheus"]
    prometheus -->|"scrapes"| prometheusEndpoint
    prometheus --> sliRules["Server-error availability / latency SLI rules"]
    sliRules --> burnAlerts["Multi-window burn-rate alerts"]
    frontend -->|"/api/sli"| backend
    backend -->|"scoped query"| prometheus
    eks --> cw["CloudWatch Logs"]

Loading

More detail: docs/architecture.md

AWS Run Results

The July 25 run deployed application release v1.2.0 to EKS 1.35, then exercised HPA and request-SLI workloads separately. The HPA run sent 18,819 requests to the bounded CPU route with no request failures, dropped iterations, or pod restarts. CPU reached 496% against a 60% target; the backend scaled from 2 to 6 replicas and returned to 2 for three spaced observations. Two 15-second samples captured new replicas becoming ready during scale-out. The timeline retains both readiness shortfalls alongside the request and restart counts.

After stabilization, the independent seven-minute read-only run completed 2,101 requests with no failed checks or dropped iterations. Both Prometheus targets and the scoped recording rules were healthy. The five-minute snapshot reported 1.0000 server-error availability and 1.0000 of eligible requests at or below 500 ms. Both deployed image scans reported zero findings, all 18 platform/runtime controls passed, and post-destroy service-native queries reached zero across 13 resource checks. The supplemental AWS tag index retained four historical mappings. The closure record marks that check inconclusive because the API returns tagged or previously tagged resources.

Historical AWS Screenshots

Live dashboard	HPA scale-out

EKS nodes	Destroy confirmation

Full gallery: docs/screenshots/aws-demo-2026-06-06

July SLI run: docs/screenshots/aws-demo-2026-07-25

Current AWS run records: docs/evidence/aws-validation-2026-07-25

Results and operational limits: docs/results.md

Technical Scope

Application

• React frontend with Vite and production Nginx image
• FastAPI backend with SQLAlchemy, Pydantic, Alembic migrations, and PostgreSQL
• Seeded data for services, incidents, incident timelines, deployments, and health checks
• REST API for service catalog, incident workflows, deployment history, dashboard summaries, and health checks
• Prometheus-compatible request count and duration telemetry

Kubernetes

• Helm chart for frontend and backend workloads
• Kubernetes Deployments, Services, ConfigMaps, Secrets, Ingress, and HPA
• Process-only liveness and database-aware readiness probes
• Resource requests and limits
• Non-root containers with read-only root filesystems and restricted capabilities
• PodDisruptionBudgets and opt-in scoped NetworkPolicies enabled by the AWS values profile
• ALB Ingress for external traffic
• Metrics Server-backed HPA validation
• Optional Prometheus Operator ServiceMonitor and PrometheusRule resources

AWS

• VPC, public/private/database subnets, route tables, Internet Gateway, and NAT Gateway
• Amazon EKS managed node group
• Amazon ECR repositories for frontend and backend images
• Amazon RDS PostgreSQL in private database subnets
• IAM roles, OIDC provider, and AWS Load Balancer Controller IRSA role
• AWS Secrets Manager database secret
• CloudWatch log groups and RDS CPU alarm

CI/CD And Quality Gates

Repository rules require pull requests and both CI workflows before merging to main, and block force pushes.

• Backend tests
• Frontend behavior tests and production build
• Docker image builds
• Helm lint and Kubernetes manifest validation
• Prometheus recording-rule unit tests and CRD-aware observability manifest validation
• SLI collector tests and k6 script syntax validation
• Terraform format and validate
• Terraform cost/security guardrail script
• Blocking Checkov Terraform scan with resource-level accepted exceptions
• ECR critical/high vulnerability gate and immutable image tags
• Manual ECR publish gate and VPC-connected EKS rollout

Repository Structure

.
├── backend/                 # FastAPI, SQLAlchemy, Alembic, tests, Dockerfile
├── frontend/                # React, Vite, Nginx production image
├── infra/                   # Terraform AWS foundation
├── charts/                  # Helm chart for EKS deployment
├── load-tests/              # k6 HPA load test
├── docs/                    # Architecture, deployment, runbooks, run notes
├── .github/workflows/       # CI/CD and Terraform validation
├── docker-compose.yml       # Local development stack
└── docker-compose.prod.yml  # Nginx-backed local release stack

Local Development

Prerequisite:

• Docker Desktop running

Start the local stack:

docker compose up -d --build
docker compose ps

Expected local endpoints:

• PostgreSQL: localhost:5432
• FastAPI: http://localhost:8000
• React/Vite: http://localhost:5173

Verify the backend:

curl http://localhost:8000/health
curl http://localhost:8000/live
curl http://localhost:8000/ready
curl http://localhost:8000/summary
curl http://localhost:8000/sli
curl http://localhost:8000/metrics

/summary returns the JSON used by the React dashboard. /metrics returns Prometheus exposition text and is blocked from the public /api/* Nginx proxy in the production image.

Run tests and frontend build:

docker compose exec backend pytest -q
docker compose exec frontend npm test
docker compose exec frontend npm run build

Expected:

16 passed
2 tests passed
✓ built

Run migrations manually when developing outside Docker:

cd backend
alembic upgrade head

Nginx-Backed Local Docker

This stack mirrors the container pattern used for EKS: React builds into static assets, Nginx serves the frontend, and /api/* proxies to FastAPI.

docker compose -f docker-compose.prod.yml up -d --build
docker compose -f docker-compose.prod.yml ps

Open:

http://localhost:8080

Verify the Nginx API proxy:

docker compose -f docker-compose.prod.yml exec frontend wget -qO- http://127.0.0.1/api/live
docker compose -f docker-compose.prod.yml exec frontend wget -qO- http://127.0.0.1/api/ready
docker compose -f docker-compose.prod.yml exec frontend wget -qO- http://127.0.0.1/api/summary

Terraform Validation

Terraform defines the AWS foundation for short-lived EKS runs:

• VPC and subnets
• EKS cluster and managed node group
• ECR repositories
• RDS PostgreSQL
• Secrets Manager database secret
• IAM roles and policies
• AWS Load Balancer Controller IRSA role
• CloudWatch log groups and alarm

Validate without deploying:

terraform -chdir=infra init -backend=false
terraform -chdir=infra fmt -check -recursive
terraform -chdir=infra validate

Expected:

Success! The configuration is valid.

Do not run terraform apply until the deploy-day checklist is complete.

Terraform state modes:

• Short-lived single-operator run: local state, with *.tfstate ignored by git
• Team-style repeatable deployment: optional S3 backend using infra/backend.tf.example

Remote state notes: docs/terraform-state.md

Helm Validation

If Helm is installed:

helm lint charts/cloudops-sre-platform -f charts/cloudops-sre-platform/values-aws-example.yaml
helm template cloudops charts/cloudops-sre-platform -f charts/cloudops-sre-platform/values-aws-example.yaml --namespace cloudops

The Prometheus Operator resources are disabled by default. Validate the observability overlay separately:

helm lint charts/cloudops-sre-platform \
  -f charts/cloudops-sre-platform/values-aws-example.yaml \
  -f charts/cloudops-sre-platform/values-observability.yaml

If Helm is not installed:

docker run --rm \
  -v "$PWD:/workspace" \
  -w /workspace \
  alpine/helm:3.15.4 lint charts/cloudops-sre-platform \
  -f charts/cloudops-sre-platform/values-aws-example.yaml

Expected:

1 chart(s) linted, 0 chart(s) failed

HPA Load Test

The /api/demo/cpu endpoint is disabled by default. For the local Nginx-backed stack, enable it only for this exercise:

DEMO_MODE=true docker compose -f docker-compose.prod.yml up -d --build

For EKS, enable backend.env.demoMode=true only for the controlled exercise described in docs/hpa-demo.md.

Smoke-test the CPU demo path locally:

docker run --rm \
  -e BASE_URL="http://host.docker.internal:8080" \
  -e TARGET_PATH="/api/demo/cpu" \
  -e CPU_DURATION_MS="10" \
  -e SMOKE_TEST="true" \
  -v "$PWD/load-tests:/scripts" \
  grafana/k6:0.54.0 run /scripts/k6-load-test.js

Expected:

checks: 100%

Restart the stack without DEMO_MODE=true after the exercise.

Full HPA runbook: docs/hpa-demo.md

AWS Run And Cleanup

This project is intentionally designed for short-lived AWS runs. The expensive resources are not meant to stay online overnight.

Cost-bearing resources include:

• EKS cluster and worker nodes
• RDS PostgreSQL
• NAT Gateway
• Application Load Balancer
• CloudWatch log ingestion

Before deploying:

1. Run local validation.
2. Confirm AWS account and region.
3. Review terraform plan.
4. Apply only when ready to capture run notes and screenshots.
5. Destroy the same day.

Run checklist: docs/aws-demo-checklist.md

Run summary: docs/aws-demo-run.md

Cost control: docs/cost-control.md

Documentation

• Architecture
• Deployment
• Security Architecture
• AWS Add-ons
• Project Results
• AWS Run Summary
• CI/CD
• HPA Demo
• Observability
• Runbook
• RDS Connectivity And Secret Rotation
• Availability Burn-Rate Response
• External Secrets Sync
• Grafana SLI Runbook
• Cost Control
• AWS Run Checklist
• Changelog

Operational Limits

The current AWS run covers one short SLI window and one HPA exercise. The database recovery drill runs in Docker Compose. These runs do not cover long-window SLO compliance, managed database recovery, or node and Availability Zone failures.

A persistent environment also requires TLS and authentication, managed secret synchronization, backup and restore tests, and an explicit operating-cost owner.