Pidoko257 · i-amdaveee · Jul 30, 2026
diff --git a/README.md b/README.md
@@ -389,6 +389,15 @@ terraform plan -var-file=environments/production.tfvars
 terraform apply
 ```
 
+## 🛠️ Operations & Incident Response
+
+- **Deployment & rollback**: [docs/BRIDGE_DEPLOYMENT_RUNBOOK.md](docs/BRIDGE_DEPLOYMENT_RUNBOOK.md)
+- **Incident runbooks**: [docs/runbooks/](docs/runbooks/README.md) — step-by-step
+  diagnosis and mitigation for the top production incidents (provider down,
+  database index bloat, high API latency, queue backlog, Redis outage, Horizon
+  degradation, DB pool exhaustion, replica lag, ledger imbalance, elevated
+  error rate).
+
 ## 🤝 Contributing
 
 We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md).

diff --git a/docs/BRIDGE_DOCUMENTATION_INDEX.md b/docs/BRIDGE_DOCUMENTATION_INDEX.md
@@ -310,6 +310,7 @@ Plus complete working code in JavaScript and Python.
 
 **For Operational Questions:**
 - [BRIDGE_DEPLOYMENT_RUNBOOK.md](./BRIDGE_DEPLOYMENT_RUNBOOK.md) - Deployment & ops
+- [runbooks/](./runbooks/README.md) - Incident runbooks for common production issues (provider down, DB bloat, latency, queue backlog, and more)
 - [BRIDGE_API_EXAMPLES.md](./BRIDGE_API_EXAMPLES.md) - API & integration
 
 **For Code Questions:**

diff --git a/docs/runbooks/01-provider-down.md b/docs/runbooks/01-provider-down.md
@@ -0,0 +1,120 @@
+# Runbook 01 — Mobile Money Provider Down
+
+**Severity:** P2 (P1 if all providers for a country are down) · **Owner:** On-call
+
+A mobile money provider (MTN MoMo, Airtel Money, Orange Money) is failing or
+timing out. ProxyPay's circuit breaker has opened (or is flapping), so
+deposits/payouts through that provider are being rejected or failed over.
+
+---
+
+## Symptoms
+
+- Alert: `provider_circuit_breaker_state{provider="..."} == 1` (open).
+- Rising `provider_failover_total` / `provider_failover_alerts_total`.
+- `transaction_errors_total{error_type="provider_error"}` spiking for one provider.
+- Users in one country report deposits/withdrawals failing or stuck.
+- `provider_response_time_seconds` climbing before the breaker trips.
+
+## How the breaker works (context)
+
+- Opens after **`healthCheck.failureThreshold` = 3** consecutive failures
+  (`HEALTH_CHECK_FAILURE_THRESHOLD`).
+- Stays open for **`healthCheck.openDurationMs` = 60000 ms** (1 min), then
+  moves to half-open and probes with a single request.
+- `provider_circuit_breaker_state`: `0`=closed (healthy), `1`=open (failing),
+  `2`=half-open (probing).
+
+---
+
+## Diagnose
+
+```bash
+# 1. Which providers are affected, and breaker state?
+curl -s localhost:3000/metrics | grep -E 'provider_circuit_breaker_state|provider_failover_total'
+
+# 2. Per-provider latency and error breakdown
+curl -s localhost:3000/metrics | grep -E 'provider_response_time_seconds|transaction_errors_total'
+```
+
+Grafana / Loki (see `../observability.md`):
+
+```logql
+{container="proxypay_app"} | json | error_type="provider_error"
+```
+
+Decide the root cause:
+
+| Signal | Likely cause |
+|--------|--------------|
+| One provider 5xx / timeouts, others fine | Provider-side outage |
+| All providers failing at once | Our egress / network / DNS / credentials |
+| 401/403 from provider | Expired API credentials or token |
+| Breaker flapping open↔closed | Provider degraded (partial), or threshold too tight |
+
+Confirm provider-side outage independently:
+- Check the provider's status page / partner portal.
+- `curl` the provider health/sandbox endpoint directly from a bridge host.
+
+---
+
+## Mitigate
+
+**If it's a single provider outage (most common):**
+
+1. Let the circuit breaker do its job — it's already failing fast and failing
+   over. Confirm failover target providers are healthy (breaker `state=0`).
+2. If a country has an alternate provider, ensure routing prefers the healthy
+   one. If not, put deposits/withdrawals for that provider into a queued/retry
+   state rather than hard-failing users where possible.
+3. Post user-facing status: "<Provider> transactions are delayed."
+
+**If it's our side (all providers failing):**
+
+1. Verify outbound network + DNS from a bridge pod:
+   ```bash
+   kubectl exec -it deploy/proxypay -- sh -c 'curl -sS -o /dev/null -w "%{http_code}\n" https://<provider-host>/'
+   ```
+2. Check credentials/secrets have not expired or rotated:
+   ```bash
+   kubectl get secret proxypay-secrets -o jsonpath='{.data}' | jq 'keys'   # keys only, never values
+   ```
+3. If credentials expired, rotate and roll pods (see `scripts/rotate-keys.ts`).
+
+**Do not** manually force the breaker closed against a genuinely-down provider —
+that just converts fast failures into slow ones and floods retries.
+
+---
+
+## Recover
+
+1. When the provider recovers, the breaker moves open → half-open → closed
+   automatically after `openDurationMs`. Watch `provider_circuit_breaker_state`
+   return to `0`.
+2. Reprocess anything that was parked. Inspect and retry failed jobs in
+   Bull-Board at `/admin/queues`, or via the admin CLI:
+   ```bash
+   npm run momo-cli -- --help    # discover retry subcommands
+   ```
+3. Confirm `transaction_total{status="success"}` recovers for that provider.
+
+---
+
+## Verify
+
+- [ ] `provider_circuit_breaker_state == 0` for the provider.
+- [ ] `transaction_errors_total{error_type="provider_error"}` flat again.
+- [ ] A test deposit + payout through the provider succeeds.
+- [ ] No orphaned/stuck transactions (spot check + see runbook 09 if unsure).
+
+---
+
+## Post-incident
+
+- If the breaker flapped, review `failureThreshold` / `openDurationMs` — a
+  degraded (not down) provider may need a longer open window.
+- If credentials expired unnoticed, add/verify a cert/credential expiry alert
+  (`npm run check-cert`).
+- File provider-side incident reference; track their RCA.
+- **Related:** [04 Queue backlog](./04-queue-backlog.md) (parked payouts),
+  [10 Elevated error rate](./10-elevated-error-rate.md).
diff --git a/docs/runbooks/02-database-index-bloat.md b/docs/runbooks/02-database-index-bloat.md
@@ -0,0 +1,114 @@
+# Runbook 02 — Database Index Bloat & Slow Queries
+
+**Severity:** P3 (P2 if latency breaches SLA on the deposit/withdraw path) · **Owner:** On-call
+
+Postgres index bloat (dead tuples accumulating in indexes) is degrading query
+performance. Symptoms overlap with high API latency (runbook 03) — this runbook
+covers the *database* root cause.
+
+---
+
+## Symptoms
+
+- Rising `slow_query` log entries (threshold `SLOW_QUERY_THRESHOLD_MS`, default 1000 ms).
+- API P99 climbing while CPU/traffic are roughly flat.
+- `db_replica_lag_seconds` rising (bloat inflates replication work).
+- Growing disk usage on the DB volume without a matching data-growth reason.
+
+---
+
+## Diagnose
+
+```bash
+# 1. What queries are slow? (structured JSON logs)
+```
+```logql
+{container="proxypay_app"} | json | type="slow_query"
+```
+
+```bash
+# 2. Audit indexes — unused, redundant, and bloated
+npm run audit:indexes -- --verbose
+```
+
+In `psql`, confirm bloat and check for missing index maintenance:
+
+```sql
+-- Top tables/indexes by dead tuples
+SELECT relname, n_dead_tup, n_live_tup,
+       round(n_dead_tup::numeric / NULLIF(n_live_tup,0), 3) AS dead_ratio,
+       last_autovacuum
+FROM pg_stat_user_tables
+ORDER BY n_dead_tup DESC
+LIMIT 20;
+
+-- Index sizes (largest first)
+SELECT indexrelname, pg_size_pretty(pg_relation_size(indexrelid)) AS size, idx_scan
+FROM pg_stat_user_indexes
+ORDER BY pg_relation_size(indexrelid) DESC
+LIMIT 20;
+
+-- Is a query using the expected index?
+EXPLAIN (ANALYZE, BUFFERS) <the slow query>;
+```
+
+Interpretation:
+- High `dead_ratio` + old `last_autovacuum` → autovacuum falling behind.
+- `idx_scan = 0` over a long window → unused index (write overhead, drop candidate).
+- Two indexes covering the same columns → redundant.
+
+---
+
+## Mitigate
+
+1. **Reindex bloated indexes** — non-blocking (`REINDEX CONCURRENTLY`), safe to
+   run online but prefer a low-traffic window:
+   ```bash
+   npm run reindex:bloated-indexes
+   ```
+   This finds bloated, eligible indexes and rebuilds them concurrently.
+
+2. **If one query is hot and unindexed**, add the index concurrently:
+   ```sql
+   CREATE INDEX CONCURRENTLY idx_<table>_<cols> ON <table> (<cols>);
+   ```
+
+3. **If autovacuum is behind** on a specific table, kick it manually:
+   ```sql
+   VACUUM (ANALYZE) <table>;
+   ```
+
+Avoid a plain `REINDEX` (non-concurrent) or `VACUUM FULL` on live tables —
+both take heavy locks and will cause an outage.
+
+---
+
+## Recover
+
+1. Re-run the audit to confirm bloat is reduced:
+   ```bash
+   npm run audit:indexes
+   ```
+2. Drop genuinely-unused indexes only after confirming across a full traffic
+   cycle (weekday + weekend); use the drop SQL the audit emits.
+3. Watch `db_replica_lag_seconds` return to baseline.
+
+---
+
+## Verify
+
+- [ ] `slow_query` log volume back to baseline.
+- [ ] `EXPLAIN ANALYZE` on the previously-slow query shows expected index usage.
+- [ ] API P99 (`http_request_duration_seconds`) recovered — see runbook 03.
+- [ ] Replica lag normal.
+
+---
+
+## Post-incident
+
+- If bloat recurs, schedule `reindex:bloated-indexes` as a cron job in a
+  low-traffic window (the script is built for this).
+- Tune autovacuum for hot tables (`autovacuum_vacuum_scale_factor`).
+- Add the offending query pattern to load tests so regressions surface early.
+- **Related:** [03 High API latency](./03-high-api-latency.md),
+  [08 Read-replica lag](./08-replica-lag.md).
diff --git a/docs/runbooks/03-high-api-latency.md b/docs/runbooks/03-high-api-latency.md
@@ -0,0 +1,105 @@
+# Runbook 03 — High API Latency
+
+**Severity:** P2 (P1 if requests are timing out on the deposit/withdraw path) · **Owner:** On-call
+
+API responses are slow. This runbook isolates *where* the latency is —
+application, database, cache, downstream provider, or Stellar — and mitigates.
+
+---
+
+## Symptoms
+
+- Alert on P99 `http_request_duration_seconds` (buckets top out at 10 s).
+- Users report slow API/app responses; possible request timeouts.
+- `active_connections` climbing (requests piling up).
+- Possible knock-on: queue backlog (runbook 04), pool exhaustion (runbook 07).
+
+---
+
+## Diagnose
+
+```bash
+# 1. Confirm latency and see which routes/status codes are slow
+curl -s localhost:3000/metrics | grep -E 'http_request_duration_seconds|active_connections'
+```
+
+P99 latency by route in Grafana (Prometheus):
+
+```promql
+histogram_quantile(0.99,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))
+```
+
+Or from logs (see `../observability.md`):
+
+```logql
+quantile_over_time(0.99, {container="proxypay_app"} | json | unwrap duration [5m])
+```
+
+Localize the bottleneck — check each layer:
+
+| Layer | Check | Points to |
+|-------|-------|-----------|
+| DB | `slow_query` logs; `db_replica_lag_seconds` | Runbook 02 / 07 / 08 |
+| Cache | `cache_hit_ratio` dropping, `cache_misses_total` up | Redis / runbook 05 |
+| Provider | `provider_response_time_seconds` high | Runbook 01 |
+| Stellar | `horizon_node_health`, `horizon_node_failures_total` | Runbook 06 |
+| App/host | Node event-loop lag, CPU, memory (default metrics) | Scale out / profile |
+
+```bash
+# Readiness confirms DB + Redis reachability quickly
+curl -s localhost:3000/ready | jq
+```
+
+---
+
+## Mitigate
+
+Act on whichever layer the diagnosis implicates:
+
+- **Downstream (provider/Stellar) slow** → follow runbook 01 / 06; the circuit
+  breaker and Horizon failover should shed load. Confirm they're engaging.
+- **DB slow** → runbook 02 (bloat/slow query) or 07 (pool). A hot missing index
+  is the most common cause of a sudden P99 jump.
+- **Cache cold/down** → runbook 05; a Redis problem turns cache hits into DB
+  reads and cascades latency everywhere.
+- **App capacity** → scale horizontally. HPA targets 80% CPU, min 2 / max 10
+  replicas (`k8s/hpa.yaml`):
+  ```bash
+  kubectl get hpa proxypay-hpa
+  kubectl scale deploy/proxypay --replicas=<n>   # temporary manual bump
+  ```
+- **Runaway/expensive endpoint** → rate-limit or temporarily disable the
+  offending route if it's non-critical.
+
+Request timeouts are enforced globally (`globalTimeout`); confirm they're not
+set so low they're manufacturing failures under load (see `../REQUEST_TIMEOUTS.md`).
+
+---
+
+## Recover
+
+1. Once the implicated layer is fixed, watch P99 return under SLA.
+2. Scale replicas back to baseline after load subsides (avoid leaving manual
+   overrides that fight the HPA).
+3. Drain any backlog that built up (runbook 04).
+
+---
+
+## Verify
+
+- [ ] P99 `http_request_duration_seconds` back under SLA.
+- [ ] `active_connections` at baseline.
+- [ ] `/ready` returns 200 with DB + Redis `ok`.
+- [ ] No sustained queue backlog.
+
+---
+
+## Post-incident
+
+- Add a Grafana panel/alert for the specific route if it was a single endpoint.
+- If capacity-driven, revisit HPA thresholds and load-test headroom
+  (`npm run test:load`).
+- **Related:** [02](./02-database-index-bloat.md), [04](./04-queue-backlog.md),
+  [05](./05-redis-outage.md), [06](./06-stellar-horizon-degraded.md),
+  [07](./07-db-pool-exhaustion.md).