From 10d44134f86f835911e4f36288910327be47a6f0 Mon Sep 17 00:00:00 2001 From: Davee Date: Thu, 30 Jul 2026 13:17:13 +0100 Subject: [PATCH] docs: add production incident runbooks (#287) Add docs/runbooks/ with an index plus 10 incident runbooks covering the top production issues: provider down, database index bloat, high API latency, queue backlog, Redis outage, Horizon degradation, DB pool exhaustion, replica lag, ledger imbalance, and elevated error rate. Each runbook is command-driven (symptoms -> diagnose -> mitigate -> recover -> verify -> post-incident) and grounded in the real codebase: metric names from src/utils/metrics.ts, health/metrics endpoints from src/index.ts, scripts from package.json, KEDA/HPA thresholds from k8s/, and the Grafana/Loki queries in docs/observability.md. Linked from docs/BRIDGE_DOCUMENTATION_INDEX.md and README.md. --- README.md | 9 ++ docs/BRIDGE_DOCUMENTATION_INDEX.md | 1 + docs/runbooks/01-provider-down.md | 120 ++++++++++++++++ docs/runbooks/02-database-index-bloat.md | 114 +++++++++++++++ docs/runbooks/03-high-api-latency.md | 105 ++++++++++++++ docs/runbooks/04-queue-backlog.md | 109 +++++++++++++++ docs/runbooks/05-redis-outage.md | 103 ++++++++++++++ docs/runbooks/06-stellar-horizon-degraded.md | 99 +++++++++++++ docs/runbooks/07-db-pool-exhaustion.md | 110 +++++++++++++++ docs/runbooks/08-replica-lag.md | 112 +++++++++++++++ docs/runbooks/09-ledger-imbalance.md | 119 ++++++++++++++++ docs/runbooks/10-elevated-error-rate.md | 108 +++++++++++++++ docs/runbooks/README.md | 138 +++++++++++++++++++ 13 files changed, 1247 insertions(+) create mode 100644 docs/runbooks/01-provider-down.md create mode 100644 docs/runbooks/02-database-index-bloat.md create mode 100644 docs/runbooks/03-high-api-latency.md create mode 100644 docs/runbooks/04-queue-backlog.md create mode 100644 docs/runbooks/05-redis-outage.md create mode 100644 docs/runbooks/06-stellar-horizon-degraded.md create mode 100644 docs/runbooks/07-db-pool-exhaustion.md create mode 100644 docs/runbooks/08-replica-lag.md create mode 100644 docs/runbooks/09-ledger-imbalance.md create mode 100644 docs/runbooks/10-elevated-error-rate.md create mode 100644 docs/runbooks/README.md diff --git a/README.md b/README.md index 3f4ff90a..65e0bad9 100644 --- a/README.md +++ b/README.md @@ -389,6 +389,15 @@ terraform plan -var-file=environments/production.tfvars terraform apply ``` +## πŸ› οΈ Operations & Incident Response + +- **Deployment & rollback**: [docs/BRIDGE_DEPLOYMENT_RUNBOOK.md](docs/BRIDGE_DEPLOYMENT_RUNBOOK.md) +- **Incident runbooks**: [docs/runbooks/](docs/runbooks/README.md) β€” step-by-step + diagnosis and mitigation for the top production incidents (provider down, + database index bloat, high API latency, queue backlog, Redis outage, Horizon + degradation, DB pool exhaustion, replica lag, ledger imbalance, elevated + error rate). + ## 🀝 Contributing We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md). diff --git a/docs/BRIDGE_DOCUMENTATION_INDEX.md b/docs/BRIDGE_DOCUMENTATION_INDEX.md index 677aeac7..4d7f3f25 100644 --- a/docs/BRIDGE_DOCUMENTATION_INDEX.md +++ b/docs/BRIDGE_DOCUMENTATION_INDEX.md @@ -310,6 +310,7 @@ Plus complete working code in JavaScript and Python. **For Operational Questions:** - [BRIDGE_DEPLOYMENT_RUNBOOK.md](./BRIDGE_DEPLOYMENT_RUNBOOK.md) - Deployment & ops +- [runbooks/](./runbooks/README.md) - Incident runbooks for common production issues (provider down, DB bloat, latency, queue backlog, and more) - [BRIDGE_API_EXAMPLES.md](./BRIDGE_API_EXAMPLES.md) - API & integration **For Code Questions:** diff --git a/docs/runbooks/01-provider-down.md b/docs/runbooks/01-provider-down.md new file mode 100644 index 00000000..cc2ee0f7 --- /dev/null +++ b/docs/runbooks/01-provider-down.md @@ -0,0 +1,120 @@ +# Runbook 01 β€” Mobile Money Provider Down + +**Severity:** P2 (P1 if all providers for a country are down) Β· **Owner:** On-call + +A mobile money provider (MTN MoMo, Airtel Money, Orange Money) is failing or +timing out. ProxyPay's circuit breaker has opened (or is flapping), so +deposits/payouts through that provider are being rejected or failed over. + +--- + +## Symptoms + +- Alert: `provider_circuit_breaker_state{provider="..."} == 1` (open). +- Rising `provider_failover_total` / `provider_failover_alerts_total`. +- `transaction_errors_total{error_type="provider_error"}` spiking for one provider. +- Users in one country report deposits/withdrawals failing or stuck. +- `provider_response_time_seconds` climbing before the breaker trips. + +## How the breaker works (context) + +- Opens after **`healthCheck.failureThreshold` = 3** consecutive failures + (`HEALTH_CHECK_FAILURE_THRESHOLD`). +- Stays open for **`healthCheck.openDurationMs` = 60000 ms** (1 min), then + moves to half-open and probes with a single request. +- `provider_circuit_breaker_state`: `0`=closed (healthy), `1`=open (failing), + `2`=half-open (probing). + +--- + +## Diagnose + +```bash +# 1. Which providers are affected, and breaker state? +curl -s localhost:3000/metrics | grep -E 'provider_circuit_breaker_state|provider_failover_total' + +# 2. Per-provider latency and error breakdown +curl -s localhost:3000/metrics | grep -E 'provider_response_time_seconds|transaction_errors_total' +``` + +Grafana / Loki (see `../observability.md`): + +```logql +{container="proxypay_app"} | json | error_type="provider_error" +``` + +Decide the root cause: + +| Signal | Likely cause | +|--------|--------------| +| One provider 5xx / timeouts, others fine | Provider-side outage | +| All providers failing at once | Our egress / network / DNS / credentials | +| 401/403 from provider | Expired API credentials or token | +| Breaker flapping open↔closed | Provider degraded (partial), or threshold too tight | + +Confirm provider-side outage independently: +- Check the provider's status page / partner portal. +- `curl` the provider health/sandbox endpoint directly from a bridge host. + +--- + +## Mitigate + +**If it's a single provider outage (most common):** + +1. Let the circuit breaker do its job β€” it's already failing fast and failing + over. Confirm failover target providers are healthy (breaker `state=0`). +2. If a country has an alternate provider, ensure routing prefers the healthy + one. If not, put deposits/withdrawals for that provider into a queued/retry + state rather than hard-failing users where possible. +3. Post user-facing status: " transactions are delayed." + +**If it's our side (all providers failing):** + +1. Verify outbound network + DNS from a bridge pod: + ```bash + kubectl exec -it deploy/proxypay -- sh -c 'curl -sS -o /dev/null -w "%{http_code}\n" https:///' + ``` +2. Check credentials/secrets have not expired or rotated: + ```bash + kubectl get secret proxypay-secrets -o jsonpath='{.data}' | jq 'keys' # keys only, never values + ``` +3. If credentials expired, rotate and roll pods (see `scripts/rotate-keys.ts`). + +**Do not** manually force the breaker closed against a genuinely-down provider β€” +that just converts fast failures into slow ones and floods retries. + +--- + +## Recover + +1. When the provider recovers, the breaker moves open β†’ half-open β†’ closed + automatically after `openDurationMs`. Watch `provider_circuit_breaker_state` + return to `0`. +2. Reprocess anything that was parked. Inspect and retry failed jobs in + Bull-Board at `/admin/queues`, or via the admin CLI: + ```bash + npm run momo-cli -- --help # discover retry subcommands + ``` +3. Confirm `transaction_total{status="success"}` recovers for that provider. + +--- + +## Verify + +- [ ] `provider_circuit_breaker_state == 0` for the provider. +- [ ] `transaction_errors_total{error_type="provider_error"}` flat again. +- [ ] A test deposit + payout through the provider succeeds. +- [ ] No orphaned/stuck transactions (spot check + see runbook 09 if unsure). + +--- + +## Post-incident + +- If the breaker flapped, review `failureThreshold` / `openDurationMs` β€” a + degraded (not down) provider may need a longer open window. +- If credentials expired unnoticed, add/verify a cert/credential expiry alert + (`npm run check-cert`). +- File provider-side incident reference; track their RCA. +- **Related:** [04 Queue backlog](./04-queue-backlog.md) (parked payouts), + [10 Elevated error rate](./10-elevated-error-rate.md). diff --git a/docs/runbooks/02-database-index-bloat.md b/docs/runbooks/02-database-index-bloat.md new file mode 100644 index 00000000..49b0d710 --- /dev/null +++ b/docs/runbooks/02-database-index-bloat.md @@ -0,0 +1,114 @@ +# Runbook 02 β€” Database Index Bloat & Slow Queries + +**Severity:** P3 (P2 if latency breaches SLA on the deposit/withdraw path) Β· **Owner:** On-call + +Postgres index bloat (dead tuples accumulating in indexes) is degrading query +performance. Symptoms overlap with high API latency (runbook 03) β€” this runbook +covers the *database* root cause. + +--- + +## Symptoms + +- Rising `slow_query` log entries (threshold `SLOW_QUERY_THRESHOLD_MS`, default 1000 ms). +- API P99 climbing while CPU/traffic are roughly flat. +- `db_replica_lag_seconds` rising (bloat inflates replication work). +- Growing disk usage on the DB volume without a matching data-growth reason. + +--- + +## Diagnose + +```bash +# 1. What queries are slow? (structured JSON logs) +``` +```logql +{container="proxypay_app"} | json | type="slow_query" +``` + +```bash +# 2. Audit indexes β€” unused, redundant, and bloated +npm run audit:indexes -- --verbose +``` + +In `psql`, confirm bloat and check for missing index maintenance: + +```sql +-- Top tables/indexes by dead tuples +SELECT relname, n_dead_tup, n_live_tup, + round(n_dead_tup::numeric / NULLIF(n_live_tup,0), 3) AS dead_ratio, + last_autovacuum +FROM pg_stat_user_tables +ORDER BY n_dead_tup DESC +LIMIT 20; + +-- Index sizes (largest first) +SELECT indexrelname, pg_size_pretty(pg_relation_size(indexrelid)) AS size, idx_scan +FROM pg_stat_user_indexes +ORDER BY pg_relation_size(indexrelid) DESC +LIMIT 20; + +-- Is a query using the expected index? +EXPLAIN (ANALYZE, BUFFERS) ; +``` + +Interpretation: +- High `dead_ratio` + old `last_autovacuum` β†’ autovacuum falling behind. +- `idx_scan = 0` over a long window β†’ unused index (write overhead, drop candidate). +- Two indexes covering the same columns β†’ redundant. + +--- + +## Mitigate + +1. **Reindex bloated indexes** β€” non-blocking (`REINDEX CONCURRENTLY`), safe to + run online but prefer a low-traffic window: + ```bash + npm run reindex:bloated-indexes + ``` + This finds bloated, eligible indexes and rebuilds them concurrently. + +2. **If one query is hot and unindexed**, add the index concurrently: + ```sql + CREATE INDEX CONCURRENTLY idx__ ON
(); + ``` + +3. **If autovacuum is behind** on a specific table, kick it manually: + ```sql + VACUUM (ANALYZE)
; + ``` + +Avoid a plain `REINDEX` (non-concurrent) or `VACUUM FULL` on live tables β€” +both take heavy locks and will cause an outage. + +--- + +## Recover + +1. Re-run the audit to confirm bloat is reduced: + ```bash + npm run audit:indexes + ``` +2. Drop genuinely-unused indexes only after confirming across a full traffic + cycle (weekday + weekend); use the drop SQL the audit emits. +3. Watch `db_replica_lag_seconds` return to baseline. + +--- + +## Verify + +- [ ] `slow_query` log volume back to baseline. +- [ ] `EXPLAIN ANALYZE` on the previously-slow query shows expected index usage. +- [ ] API P99 (`http_request_duration_seconds`) recovered β€” see runbook 03. +- [ ] Replica lag normal. + +--- + +## Post-incident + +- If bloat recurs, schedule `reindex:bloated-indexes` as a cron job in a + low-traffic window (the script is built for this). +- Tune autovacuum for hot tables (`autovacuum_vacuum_scale_factor`). +- Add the offending query pattern to load tests so regressions surface early. +- **Related:** [03 High API latency](./03-high-api-latency.md), + [08 Read-replica lag](./08-replica-lag.md). diff --git a/docs/runbooks/03-high-api-latency.md b/docs/runbooks/03-high-api-latency.md new file mode 100644 index 00000000..bec13cad --- /dev/null +++ b/docs/runbooks/03-high-api-latency.md @@ -0,0 +1,105 @@ +# Runbook 03 β€” High API Latency + +**Severity:** P2 (P1 if requests are timing out on the deposit/withdraw path) Β· **Owner:** On-call + +API responses are slow. This runbook isolates *where* the latency is β€” +application, database, cache, downstream provider, or Stellar β€” and mitigates. + +--- + +## Symptoms + +- Alert on P99 `http_request_duration_seconds` (buckets top out at 10 s). +- Users report slow API/app responses; possible request timeouts. +- `active_connections` climbing (requests piling up). +- Possible knock-on: queue backlog (runbook 04), pool exhaustion (runbook 07). + +--- + +## Diagnose + +```bash +# 1. Confirm latency and see which routes/status codes are slow +curl -s localhost:3000/metrics | grep -E 'http_request_duration_seconds|active_connections' +``` + +P99 latency by route in Grafana (Prometheus): + +```promql +histogram_quantile(0.99, + sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)) +``` + +Or from logs (see `../observability.md`): + +```logql +quantile_over_time(0.99, {container="proxypay_app"} | json | unwrap duration [5m]) +``` + +Localize the bottleneck β€” check each layer: + +| Layer | Check | Points to | +|-------|-------|-----------| +| DB | `slow_query` logs; `db_replica_lag_seconds` | Runbook 02 / 07 / 08 | +| Cache | `cache_hit_ratio` dropping, `cache_misses_total` up | Redis / runbook 05 | +| Provider | `provider_response_time_seconds` high | Runbook 01 | +| Stellar | `horizon_node_health`, `horizon_node_failures_total` | Runbook 06 | +| App/host | Node event-loop lag, CPU, memory (default metrics) | Scale out / profile | + +```bash +# Readiness confirms DB + Redis reachability quickly +curl -s localhost:3000/ready | jq +``` + +--- + +## Mitigate + +Act on whichever layer the diagnosis implicates: + +- **Downstream (provider/Stellar) slow** β†’ follow runbook 01 / 06; the circuit + breaker and Horizon failover should shed load. Confirm they're engaging. +- **DB slow** β†’ runbook 02 (bloat/slow query) or 07 (pool). A hot missing index + is the most common cause of a sudden P99 jump. +- **Cache cold/down** β†’ runbook 05; a Redis problem turns cache hits into DB + reads and cascades latency everywhere. +- **App capacity** β†’ scale horizontally. HPA targets 80% CPU, min 2 / max 10 + replicas (`k8s/hpa.yaml`): + ```bash + kubectl get hpa proxypay-hpa + kubectl scale deploy/proxypay --replicas= # temporary manual bump + ``` +- **Runaway/expensive endpoint** β†’ rate-limit or temporarily disable the + offending route if it's non-critical. + +Request timeouts are enforced globally (`globalTimeout`); confirm they're not +set so low they're manufacturing failures under load (see `../REQUEST_TIMEOUTS.md`). + +--- + +## Recover + +1. Once the implicated layer is fixed, watch P99 return under SLA. +2. Scale replicas back to baseline after load subsides (avoid leaving manual + overrides that fight the HPA). +3. Drain any backlog that built up (runbook 04). + +--- + +## Verify + +- [ ] P99 `http_request_duration_seconds` back under SLA. +- [ ] `active_connections` at baseline. +- [ ] `/ready` returns 200 with DB + Redis `ok`. +- [ ] No sustained queue backlog. + +--- + +## Post-incident + +- Add a Grafana panel/alert for the specific route if it was a single endpoint. +- If capacity-driven, revisit HPA thresholds and load-test headroom + (`npm run test:load`). +- **Related:** [02](./02-database-index-bloat.md), [04](./04-queue-backlog.md), + [05](./05-redis-outage.md), [06](./06-stellar-horizon-degraded.md), + [07](./07-db-pool-exhaustion.md). diff --git a/docs/runbooks/04-queue-backlog.md b/docs/runbooks/04-queue-backlog.md new file mode 100644 index 00000000..d73bd7c3 --- /dev/null +++ b/docs/runbooks/04-queue-backlog.md @@ -0,0 +1,109 @@ +# Runbook 04 β€” Queue Backlog + +**Severity:** P2 (P1 if payouts are the backed-up queue and funds are stuck) Β· **Owner:** On-call + +BullMQ jobs are accumulating faster than workers can process them. Deposits, +payouts, provider balance alerts, or account-merge jobs are delayed. + +--- + +## Symptoms + +- Alert on queue depth: `total_depth` from `/health/queue/depth` staying high. +- KEDA has scaled workers to `maxReplicaCount` (20) and depth is still rising. +- Users report deposits/withdrawals "pending" longer than usual. +- `latency_ms` (age of oldest waiting jobs) growing in the depth response. + +## How scaling works (context) + +- KEDA polls `GET /health/queue/depth` every 30 s and reads `total_depth`. +- Scales the **worker** Deployment up when `total_depth > 20` per replica. +- `minReplicaCount = 1`, `maxReplicaCount = 20`, `cooldownPeriod = 60 s` + (`k8s/keda-scaled-object.yaml`). + +--- + +## Diagnose + +```bash +# 1. Current depth, per-queue breakdown, and Redis memory +curl -s localhost:3000/health/queue/depth | jq +curl -s localhost:3000/health/queue | jq + +# 2. Are workers actually scaling / running? +kubectl get scaledobject proxypay-worker-scaledobject +kubectl get deploy proxypay-worker -o wide +kubectl get pods -l app=proxypay-worker +``` + +Open **Bull-Board** at `/admin/queues` to inspect waiting/active/failed jobs +per queue and see failure reasons. + +Determine which pattern you're in: + +| Pattern | Likely cause | +|---------|--------------| +| Depth high, workers NOT scaling | KEDA / metrics-api trigger broken; check `/health/queue/depth` reachable in-cluster | +| Workers at max, still growing | Genuine overload, or a downstream dependency slow (provider/Stellar/DB) | +| Many `failed` + retrying jobs | Poison job or downstream error causing retry storms | +| One queue backed up, others fine | That queue's downstream is the bottleneck | + +--- + +## Mitigate + +1. **Workers not scaling** β€” verify KEDA can reach the endpoint and it returns + valid JSON: + ```bash + kubectl run curl --rm -it --image=curlimages/curl --restart=Never -- \ + curl -s http://proxypay-service.default.svc.cluster.local:3000/health/queue/depth + ``` + If the trigger is broken, manually scale workers as a stopgap: + ```bash + kubectl scale deploy/proxypay-worker --replicas=20 + ``` + +2. **Genuine overload** β€” workers are maxed and healthy: the bottleneck is + almost always downstream. Check and fix per: + - Provider slow/down β†’ [01](./01-provider-down.md) + - Stellar Horizon slow β†’ [06](./06-stellar-horizon-degraded.md) + - DB slow / pool exhausted β†’ [02](./02-database-index-bloat.md) / [07](./07-db-pool-exhaustion.md) + +3. **Retry storm / poison job** β€” in Bull-Board, identify the failing job(s). + Pause the queue if retries are amplifying load, fix/skip the poison job, + then resume. Do **not** blanket-delete payout jobs β€” money may be involved. + +4. **Redis memory pressure** β€” if `redis_memory_bytes` is near the limit, that + caps queue throughput; see [05](./05-redis-outage.md). + +--- + +## Recover + +1. With the bottleneck cleared, watch `total_depth` drain toward 0 and + `latency_ms` fall. +2. Reprocess failed jobs from Bull-Board once the downstream is healthy. +3. Let KEDA scale workers back down after the `cooldownPeriod`; remove any + manual `kubectl scale` override so autoscaling resumes. + +--- + +## Verify + +- [ ] `total_depth` back to baseline (near 0 waiting). +- [ ] No growing `failed` count in `/admin/queues`. +- [ ] For payout queues: every job either completed or explicitly reconciled + (cross-check with [09 Ledger imbalance](./09-ledger-imbalance.md)). +- [ ] Worker replica count returned to autoscaled baseline. + +--- + +## Post-incident + +- If the max of 20 workers was insufficient, raise `maxReplicaCount` and + re-load-test (`npm run test:load:spike-10k`). +- If a poison job caused it, add validation/guardrails so it fails fast to a + dead-letter state instead of retrying. +- Consider alerting on `latency_ms` (age) in addition to raw depth. +- **Related:** [01](./01-provider-down.md), [03](./03-high-api-latency.md), + [05](./05-redis-outage.md), [06](./06-stellar-horizon-degraded.md). diff --git a/docs/runbooks/05-redis-outage.md b/docs/runbooks/05-redis-outage.md new file mode 100644 index 00000000..3d587509 --- /dev/null +++ b/docs/runbooks/05-redis-outage.md @@ -0,0 +1,103 @@ +# Runbook 05 β€” Redis Outage / Failover + +**Severity:** P1 (Redis is a hard dependency for readiness) Β· **Owner:** On-call + eng lead + +Redis is down, unreachable, or mid-failover. Redis backs sessions, caching, +BullMQ queues, rate limiting, distributed locks, and WebSocket pub/sub β€” so an +outage degrades nearly everything. + +--- + +## Symptoms + +- `/ready` returns 503 with `redis: down` or `redis: closed`. +- `/health/lb` failing β†’ load balancer pulls the node out of rotation. +- `cache_hit_ratio` collapses; `cache_misses_total` spikes β†’ DB load rises. +- Queue processing stalls (BullMQ needs Redis) β€” see [04](./04-queue-backlog.md). +- Logs: connection refused / `READONLY You can't write against a read only replica`. + +## Context β€” Sentinel failover + +- If Sentinel is enabled, the app listens for `+switch-master` events and + force-reconnects (`src/config/redis.ts`). +- On a `READONLY` reply (connected to a replica after failover), the app forces + a failover reconnect automatically. A brief blip during promotion is expected. + +--- + +## Diagnose + +```bash +# 1. App's own view +curl -s localhost:3000/ready | jq '.checks' + +# 2. Is Redis reachable and who is master? +redis-cli -h -p ping +redis-cli -h -p info replication | grep -E 'role|master_link_status' + +# 3. Sentinel view (if used) +redis-cli -h -p 26379 sentinel masters +redis-cli -h -p 26379 sentinel get-master-addr-by-name + +# 4. Memory / eviction pressure +redis-cli -h -p info memory | grep -E 'used_memory_human|maxmemory' +``` + +| Signal | Cause | +|--------|-------| +| Master unreachable, Sentinel promoting | Failover in progress β€” usually self-heals | +| `master_link_status:down` on replicas | Replication broken | +| `used_memory` at `maxmemory`, evictions | Memory exhaustion (see redis.conf) | +| Connection refused everywhere | Redis process/cluster down | + +--- + +## Mitigate + +1. **Failover in progress** β€” the app force-reconnects on `+switch-master` / + `READONLY`. Give it up to the Sentinel `down-after-milliseconds` + promotion + window. If pods are stuck on stale connections, roll them: + ```bash + kubectl rollout restart deploy/proxypay deploy/proxypay-worker + ``` + +2. **Redis process/cluster down** β€” restart/replace the failed node. If managed + (e.g. ElastiCache), trigger failover to a healthy replica from the console. + +3. **Memory exhaustion** β€” check the eviction policy in `redis.conf`. If a queue + or key set is ballooning, identify it (`redis-cli --bigkeys`) and address the + producer. Scale Redis memory if genuinely undersized. + +4. **Protect Postgres while cache is cold** β€” cache misses now hit the DB + directly. Watch DB load and be ready to shed non-critical traffic; see + [07 DB pool exhaustion](./07-db-pool-exhaustion.md). + +--- + +## Recover + +1. Confirm a single healthy master and connected replicas + (`info replication`). +2. `/ready` returns 200 with `redis: ok` across all app pods. +3. Queues resume draining (runbook 04); `cache_hit_ratio` climbs back. + +--- + +## Verify + +- [ ] `/ready` and `/health/lb` return 200 on all pods. +- [ ] `role:master` on exactly one node; replicas `master_link_status:up`. +- [ ] `cache_hit_ratio` recovering; DB load back to baseline. +- [ ] Queue depth draining, no stuck jobs. + +--- + +## Post-incident + +- If failover was slow, review Sentinel `down-after-milliseconds` / + `failover-timeout` and app reconnect behavior. +- If memory-driven, right-size `maxmemory` and confirm eviction policy suits + cache vs. queue data (queues must not be evicted). +- Verify sessions/rate-limits degraded gracefully (no auth lockout storm). +- **Related:** [04](./04-queue-backlog.md), [03](./03-high-api-latency.md), + [07](./07-db-pool-exhaustion.md). diff --git a/docs/runbooks/06-stellar-horizon-degraded.md b/docs/runbooks/06-stellar-horizon-degraded.md new file mode 100644 index 00000000..5bc7fe10 --- /dev/null +++ b/docs/runbooks/06-stellar-horizon-degraded.md @@ -0,0 +1,99 @@ +# Runbook 06 β€” Stellar Horizon Degradation + +**Severity:** P2 (P1 if all Horizon nodes are down and settlement stops) Β· **Owner:** On-call + +Stellar Horizon (the API into the Stellar network) is slow, erroring, or down. +ProxyPay can rotate across a list of Horizon URLs; this runbook covers detecting +the failure and confirming failover. + +--- + +## Symptoms + +- `horizon_node_health{node="..."} == 0` for one or more nodes. +- `horizon_node_failures_total` / `horizon_request_failover_total` rising. +- Stellar-leg transactions slow or failing: `transaction_errors_total{error_type="stellar_error"}`. +- Deposits credited but Stellar settlement delayed (or vice-versa for withdrawals). + +## Context + +- `STELLAR_HORIZON_URL` may be a comma-separated list (primary first, then + fallbacks) for automatic node rotation/failover (`src/config/env.ts`). +- The bridge fails over to the next node on failure and records it in + `horizon_request_failover_total`. + +--- + +## Diagnose + +```bash +# 1. Per-node health and failover activity +curl -s localhost:3000/metrics | grep -E 'horizon_node_health|horizon_node_failures_total|horizon_request_failover_total' + +# 2. Hit Horizon directly to see if it's Horizon or us +curl -s https://horizon.stellar.org/ | jq '{horizon_version, core_state: .core_status, latest_ledger}' +``` + +Check independently: +- Stellar network status page / status.stellar.org. +- `core_status` / ledger advancing β€” if `latest_ledger` is stale, the network + or that node is stuck, not just slow. + +| Signal | Cause | +|--------|-------| +| One node `health=0`, failover working | Single Horizon node issue β€” expected to self-mitigate | +| All nodes `health=0` | Network-wide Horizon issue, or our egress/DNS | +| 429 from Horizon | Rate limited β€” too many requests / need own node | +| Ledger not advancing | Stellar network degradation (rare) | + +--- + +## Mitigate + +1. **Single node down** β€” confirm failover is engaging + (`horizon_request_failover_total` incrementing, other node `health=1`). No + action needed beyond monitoring. + +2. **All configured nodes unhealthy** β€” add a known-good Horizon endpoint to the + front of `STELLAR_HORIZON_URL` and roll pods: + ```bash + kubectl set env deploy/proxypay STELLAR_HORIZON_URL="https://,https://horizon.stellar.org" + kubectl rollout status deploy/proxypay + ``` + +3. **Rate limited (429)** β€” back off submission rate; if this recurs, plan a + dedicated/paid Horizon instance. Ensure fee-bumping isn't retrying too + aggressively (see `../FEE_BUMPING_IMPLEMENTATION.md`). + +4. **Network degradation** β€” if ledgers aren't advancing network-wide, pause new + Stellar submissions to avoid a pile-up of stuck transactions; queue them for + replay. Communicate delays to users. Do not double-submit. + +--- + +## Recover + +1. When Horizon recovers, `horizon_node_health` returns to `1`. +2. Replay/verify any Stellar transactions that were queued or timed out β€” + check by transaction hash on Horizon before resubmitting to avoid double + settlement. +3. Confirm deposit↔settlement pairing is intact (see [09](./09-ledger-imbalance.md)). + +--- + +## Verify + +- [ ] `horizon_node_health == 1` for the primary node. +- [ ] `transaction_errors_total{error_type="stellar_error"}` flat. +- [ ] A test Stellar payment settles end to end. +- [ ] No transactions stuck between mobile-money and Stellar legs. + +--- + +## Post-incident + +- If a single public node keeps flapping, add more fallbacks or run a private + Horizon. +- Alert on `latest_ledger` staleness, not just node health. +- **Related:** [04](./04-queue-backlog.md), [09](./09-ledger-imbalance.md), + [10](./10-elevated-error-rate.md). diff --git a/docs/runbooks/07-db-pool-exhaustion.md b/docs/runbooks/07-db-pool-exhaustion.md new file mode 100644 index 00000000..8bb1fb86 --- /dev/null +++ b/docs/runbooks/07-db-pool-exhaustion.md @@ -0,0 +1,110 @@ +# Runbook 07 β€” Database Connection Pool Exhaustion + +**Severity:** P1 (writes failing = deposits/payouts failing) Β· **Owner:** On-call + eng lead + +The Postgres connection pool (or the server's `max_connections`) is exhausted. +New queries block or fail, and `/ready` starts returning 503 on its DB check. + +--- + +## Symptoms + +- `/ready` returns 503 with `database: down`, intermittently. +- Errors: `sorry, too many clients already`, `remaining connection slots are + reserved`, or pool `timeout acquiring a connection`. +- API latency spikes then errors (see [03](./03-high-api-latency.md)). +- Often triggered by a downstream slowdown holding connections open longer. + +--- + +## Diagnose + +```bash +# 1. App readiness / DB reachability +curl -s localhost:3000/ready | jq '.checks' +``` + +In `psql`: + +```sql +-- How many connections, by state, and against the limit? +SELECT count(*) AS total, + count(*) FILTER (WHERE state = 'active') AS active, + count(*) FILTER (WHERE state = 'idle in transaction') AS idle_in_txn +FROM pg_stat_activity; + +SHOW max_connections; + +-- Longest-running / stuck queries +SELECT pid, state, now() - query_start AS duration, left(query, 120) AS query +FROM pg_stat_activity +WHERE state <> 'idle' +ORDER BY duration DESC +LIMIT 20; +``` + +| Signal | Cause | +|--------|-------| +| Many `idle in transaction` | A code path opens a txn and doesn't commit/rollback (leak) | +| Many long `active` queries | Slow queries holding connections β€” see [02](./02-database-index-bloat.md) | +| Total β‰ˆ `max_connections` | Pool sized too high, or too many app/worker replicas | +| Spike aligns with traffic | Genuine load β€” need pooling / scaling, not a leak | + +--- + +## Mitigate + +1. **Kill offending sessions** (buys headroom immediately). Prefer cancel over + terminate; never blanket-kill without reading the queries: + ```sql + -- Cancel long-running non-idle queries older than N minutes + SELECT pg_cancel_backend(pid) + FROM pg_stat_activity + WHERE state = 'active' AND now() - query_start > interval '5 minutes'; + + -- Terminate leaked idle-in-transaction sessions + SELECT pg_terminate_backend(pid) + FROM pg_stat_activity + WHERE state = 'idle in transaction' AND now() - state_change > interval '5 minutes'; + ``` + +2. **Reduce demand** β€” if a burst of replicas is the cause, the pool size Γ— + replica count may exceed `max_connections`. Scale app/worker replicas *down* + temporarily, or lower per-instance pool size and roll. + +3. **Fix the upstream cause** β€” pool exhaustion is usually a *symptom*: + - Slow queries holding connections β†’ [02](./02-database-index-bloat.md). + - Replica lag pushing reads to primary β†’ [08](./08-replica-lag.md). + - Downstream stall (provider/Stellar) holding request handlers open β†’ + [01](./01-provider-down.md) / [06](./06-stellar-horizon-degraded.md). + +4. Route eligible reads to replicas if read-replica routing is disabled + (`db_replica_read_enabled` gauge); see `../read-replica-routing.md`. + +--- + +## Recover + +1. Confirm connection count drops well below `max_connections`. +2. `/ready` returns 200 with `database: ok` across pods. +3. Return replica counts / pool sizes to baseline once stable. + +--- + +## Verify + +- [ ] `pg_stat_activity` total connections have headroom vs. `max_connections`. +- [ ] No `idle in transaction` sessions accumulating. +- [ ] `/ready` green on all pods; API latency normal. + +--- + +## Post-incident + +- If a leak: find the code path missing a commit/rollback and add a regression + test. `idle in transaction` at zero is the target. +- Size the pool deliberately: `pool_size Γ— (app + worker replicas) < max_connections` + (leave a reserve for superuser + maintenance). +- Consider a server-side pooler (PgBouncer) if replica count is elastic. +- **Related:** [02](./02-database-index-bloat.md), [03](./03-high-api-latency.md), + [08](./08-replica-lag.md). diff --git a/docs/runbooks/08-replica-lag.md b/docs/runbooks/08-replica-lag.md new file mode 100644 index 00000000..2ad874de --- /dev/null +++ b/docs/runbooks/08-replica-lag.md @@ -0,0 +1,112 @@ +# Runbook 08 β€” Read-Replica Lag + +**Severity:** P3 (P2 if stale reads cause incorrect balances/decisions) Β· **Owner:** On-call + +A Postgres read replica is lagging behind the primary. Reads routed to it return +stale data β€” users may see out-of-date balances, transaction status, or history. + +--- + +## Symptoms + +- `db_replica_lag_seconds` above threshold and climbing. +- Users report "my deposit isn't showing" / stale balances that self-correct. +- Read-after-write inconsistencies (write on primary, read from lagging replica). +- Possible cause or effect of DB load β€” see [02](./02-database-index-bloat.md) / [07](./07-db-pool-exhaustion.md). + +## Context + +- Read routing is controlled by `db_replica_read_enabled` (gauge: 1=enabled). + Metrics/report queries use `queryRead()` to leverage replicas. +- The app can run in a DR `failover` mode where writes redirect to a promoted + replica (`src/config/database.ts`). + +--- + +## Diagnose + +```bash +# 1. App's reported lag and whether replica reads are enabled +curl -s localhost:3000/metrics | grep -E 'db_replica_lag_seconds|db_replica_read_enabled' +``` + +On the **replica**: + +```sql +-- Lag in seconds (0 or NULL when caught up / no traffic) +SELECT now() - pg_last_xact_replay_timestamp() AS replay_lag; +SELECT pg_is_in_recovery(); -- should be true on a replica +``` + +On the **primary**: + +```sql +-- Per-replica send/write/flush/replay positions +SELECT client_addr, state, sent_lsn, replay_lsn, + pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replay_bytes_behind +FROM pg_stat_replication; +``` + +| Signal | Cause | +|--------|-------| +| `replay_bytes_behind` large & growing | Replica can't keep up (I/O, CPU, or a long query blocking replay) | +| Lag spikes during heavy writes | Write burst / bulk job (e.g. reindex, batch payout) | +| One replica lags, others fine | That replica's host is unhealthy | +| `state` not `streaming` | Replication broken / disconnected | + +--- + +## Mitigate + +1. **Protect correctness first.** If stale reads are causing wrong balances or + decisions, disable replica reads so traffic goes to the primary until lag + clears: + ```bash + kubectl set env deploy/proxypay DB_REPLICA_READ_ENABLED=false + ``` + (Watch `db_replica_read_enabled` drop to 0. This raises primary load β€” keep + an eye on [07](./07-db-pool-exhaustion.md).) + +2. **Find what's blocking replay** on the replica β€” a long-running read query + can pause WAL replay: + ```sql + SELECT pid, now() - query_start AS duration, left(query,120) + FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC; + ``` + Cancel the offender if safe. + +3. **Write burst** β€” if a bulk job (reindex, batch payout, backfill) is driving + lag, throttle or defer it to a low-traffic window. + +4. **Replication broken** (`state` not `streaming`) β€” check replica logs, disk + space, and network to primary; re-establish streaming / rebuild the replica + if it fell too far behind (WAL recycled). + +--- + +## Recover + +1. Watch `db_replica_lag_seconds` fall back toward 0. +2. Re-enable replica reads once caught up: + ```bash + kubectl set env deploy/proxypay DB_REPLICA_READ_ENABLED=true + ``` +3. Confirm primary load returns to baseline after re-enabling. + +--- + +## Verify + +- [ ] `db_replica_lag_seconds` at baseline (near 0). +- [ ] `pg_stat_replication.state = 'streaming'` for all replicas. +- [ ] Read-after-write consistency confirmed with a test deposit. +- [ ] Primary connection count healthy (if reads were redirected). + +--- + +## Post-incident + +- If lag came from bulk jobs, schedule them off-peak and/or throttle batch size. +- If a replica repeatedly falls behind, right-size its host (I/O in particular). +- Consider `hot_standby_feedback` / `max_standby_streaming_delay` tuning tradeoffs. +- **Related:** [02](./02-database-index-bloat.md), [07](./07-db-pool-exhaustion.md). diff --git a/docs/runbooks/09-ledger-imbalance.md b/docs/runbooks/09-ledger-imbalance.md new file mode 100644 index 00000000..aef91fab --- /dev/null +++ b/docs/runbooks/09-ledger-imbalance.md @@ -0,0 +1,119 @@ +# Runbook 09 β€” Ledger Imbalance + +**Severity:** P1 (money integrity) Β· **Owner:** On-call + eng lead + finance + +The double-entry ledger fails to balance: total debits β‰  total credits, or +reconciliation reports orphaned transactions / invalid balances. Treat any +ledger imbalance as **funds-at-risk** until proven otherwise. + +--- + +## Symptoms + +- `reconcile:ledger` reports `ledgerBalanced: false` (non-zero `difference`). +- Reconciliation lists orphaned transactions or accounts with invalid balances. +- Cross-chain mismatch: `cross_chain_anomaly_total` incrementing, or + `cross_chain_balance` gauge diverging from expected. +- Mobile-money leg completed without the matching Stellar leg (or vice-versa). + +--- + +## Diagnose + +```bash +# 1. Run reconciliation (optionally as-of a date) +npm run reconcile:ledger +npm run reconcile:ledger -- --date=2026-07-30 +``` + +The report gives: `totalDebits`, `totalCredits`, `difference`, a trial balance, +plus `issues[]` and `warnings[]`. Read them before touching anything. + +```bash +# 2. Cross-chain balance anomalies +curl -s localhost:3000/metrics | grep -E 'cross_chain_anomaly_total|cross_chain_balance' +``` + +In `psql`, localize the imbalance (adjust to schema; use the trial balance to +find which account is off): + +```sql +-- Journal entries that don't net to zero per transaction. +-- Schema: ledger_entries(debit_amount, credit_amount, transaction_id, account_id, ...) +-- (immutable double-entry table; exactly one of debit_amount/credit_amount is non-zero per row) +SELECT transaction_id, + sum(debit_amount) AS debits, + sum(credit_amount) AS credits, + sum(debit_amount) - sum(credit_amount) AS diff +FROM ledger_entries +GROUP BY transaction_id +HAVING sum(debit_amount) <> sum(credit_amount) +ORDER BY abs(sum(debit_amount) - sum(credit_amount)) DESC +LIMIT 50; +``` + +| Signal | Likely cause | +|--------|--------------| +| Single txn off | Partial write / crash mid-transaction (one leg only) | +| Off since a deploy | Regression in ledger-writing code | +| Cross-chain gauge diverges | Stellar leg settled/failed without ledger update ([06](./06-stellar-horizon-degraded.md)) | +| Many small diffs | Rounding / fee-posting bug | + +--- + +## Mitigate + +1. **Contain first.** If a code path is actively writing unbalanced entries, + pause the affected flow (e.g. stop the payout/settlement worker) so the gap + stops growing. Do **not** delete or "fix up" ledger rows manually. + ```bash + # e.g. pause the affected queue in Bull-Board (/admin/queues) or scale workers to 0 + kubectl scale deploy/proxypay-worker --replicas=0 + ``` + +2. **Preserve evidence.** Snapshot the ledger tables / take a backup before any + corrective action β€” this is a financial record. + ```bash + npm run backup:create && npm run backup:verify + ``` + +3. **Triage the specific transactions** from the query above. For each, confirm + the real-world truth on both legs: + - Mobile-money leg: provider transaction status. + - Stellar leg: transaction hash on Horizon. + +4. Engage finance/eng lead. Correcting a ledger is a **compensating-entry** + exercise (append correcting journal entries), never an in-place edit. + +--- + +## Recover + +1. Post compensating entries (via the proper ledger service path) so the books + balance, referencing the incident. Never hand-edit historical rows. +2. Resume the paused flow only after the write bug (if any) is fixed and + deployed. +3. Re-run `npm run reconcile:ledger` until `ledgerBalanced: true` with zero + unexplained `issues[]`. + +--- + +## Verify + +- [ ] `reconcile:ledger` β†’ `ledgerBalanced: true`, `difference == 0`. +- [ ] No orphaned transactions or invalid balances in the report. +- [ ] `cross_chain_anomaly_total` no longer incrementing; balances reconcile. +- [ ] Every triaged transaction matched to real provider + Stellar state. + +--- + +## Post-incident + +- Root-cause the write path: transactions must post both legs atomically + (all-or-nothing) β€” add a test proving a crash mid-write cannot leave a + half-posted entry. +- Add/confirm an automated scheduled `reconcile:ledger` with alerting on + imbalance, so this is caught in minutes, not by users. +- Full financial post-mortem with finance sign-off. +- **Related:** [06](./06-stellar-horizon-degraded.md), [01](./01-provider-down.md), + [04](./04-queue-backlog.md). diff --git a/docs/runbooks/10-elevated-error-rate.md b/docs/runbooks/10-elevated-error-rate.md new file mode 100644 index 00000000..0e61dd67 --- /dev/null +++ b/docs/runbooks/10-elevated-error-rate.md @@ -0,0 +1,108 @@ +# Runbook 10 β€” Elevated Error Rate (incl. Traffic Spike) + +**Severity:** P2 (P1 if error rate on the deposit/withdraw path is high) Β· **Owner:** On-call + +The overall error rate is elevated β€” a spike in 5xx responses and/or +transaction failures. This is often the *first* alert you get; use it to +localize which subsystem is failing, then jump to that runbook. Traffic spikes +are one common trigger and are handled here too. + +--- + +## Symptoms + +- Error-rate alert (see LogQL below) or 5xx spike in `http_requests_total`. +- `transaction_errors_total` climbing (by `error_type`). +- Possible correlated load: `active_connections` high, queue depth rising. + +--- + +## Diagnose + +```bash +# 1. Errors by status code and route +curl -s localhost:3000/metrics | grep -E 'http_requests_total|transaction_errors_total' +``` + +Error rate % (Grafana / Loki, from `../observability.md`): + +```logql +sum(rate({container="proxypay_app"} | json | level="ERROR" [5m])) +/ sum(rate({container="proxypay_app"} [5m])) * 100 +``` + +**Localize by `error_type`** on `transaction_errors_total` β€” this points +straight at the responsible runbook: + +| `error_type` / signal | Root cause β†’ runbook | +|-----------------------|----------------------| +| `provider_error` | Mobile money provider β†’ [01](./01-provider-down.md) | +| `stellar_error` | Horizon degradation β†’ [06](./06-stellar-horizon-degraded.md) | +| `exception` + DB errors | Slow queries / pool β†’ [02](./02-database-index-bloat.md) / [07](./07-db-pool-exhaustion.md) | +| Redis/cache errors | Redis outage β†’ [05](./05-redis-outage.md) | +| 5xx across all routes + high load | Traffic spike (below) | +| Errors since a deploy | Bad release β†’ **roll back** (below) | + +```bash +# 2. Did this start at a deploy? Compare error onset to rollout time. +kubectl rollout history deploy/proxypay +curl -s localhost:3000/health | jq .gitHash # currently-running build +``` + +--- + +## Mitigate + +### If it's a bad deploy +Roll back β€” fastest safe action: +```bash +kubectl rollout undo deploy/proxypay +kubectl rollout status deploy/proxypay +``` +(See `../BRIDGE_DEPLOYMENT_RUNBOOK.md` β†’ Rollback Procedures.) + +### If it's a traffic spike +1. Confirm it's load, not a bug: 5xx broad, latency up, resources saturated. +2. Scale the API tier β€” HPA targets 80% CPU (min 2 / max 10, `k8s/hpa.yaml`); + bump the ceiling or replicas if it's pinned: + ```bash + kubectl get hpa proxypay-hpa + kubectl scale deploy/proxypay --replicas= + ``` + Workers autoscale on queue depth via KEDA (max 20) β€” see [04](./04-queue-backlog.md). +3. Rate limiting is multi-layer (`express-rate-limit`, `rate-limiter-flexible`); + confirm limits are shedding abusive traffic without blocking legitimate + users. Tighten temporarily if a single client/IP is the source. +4. Protect the data tier β€” a spike cascades into DB pool ([07](./07-db-pool-exhaustion.md)) + and Redis ([05](./05-redis-outage.md)); watch both. + +### If it's one subsystem +Jump to the runbook the `error_type` table points to and mitigate there. + +--- + +## Recover + +1. Error rate returns under threshold; 5xx back to baseline. +2. Drain any backlog that accumulated ([04](./04-queue-backlog.md)). +3. Scale replicas back to baseline once load subsides; remove manual overrides. + +--- + +## Verify + +- [ ] Error rate % back under alert threshold (LogQL query above). +- [ ] `transaction_errors_total` flat across all `error_type`s. +- [ ] `/ready` green; latency and `active_connections` normal. +- [ ] Running `gitHash` is the intended build (if a rollback occurred). + +--- + +## Post-incident + +- Bad deploy β†’ add the failure to CI (test/load) so it can't ship again; + review canary/staging coverage in `../BRIDGE_DEPLOYMENT_RUNBOOK.md`. +- Traffic spike β†’ revisit HPA/KEDA headroom and rate-limit thresholds; re-run + `npm run test:load:spike-10k` to validate capacity. +- Ensure the error-rate alert links directly to this runbook. +- **Related:** all subsystem runbooks (01–09); this is the fan-out point. diff --git a/docs/runbooks/README.md b/docs/runbooks/README.md new file mode 100644 index 00000000..5049d587 --- /dev/null +++ b/docs/runbooks/README.md @@ -0,0 +1,138 @@ +# ProxyPay Production Runbooks + +**Status:** Production | **Last Updated:** July 2026 + +Operational runbooks for the most common production incidents on the ProxyPay +Mobile Money ↔ Stellar bridge. Each runbook is self-contained: symptoms β†’ +diagnosis β†’ mitigation β†’ recovery β†’ post-incident. + +> For **deployment and rollback** procedures, see +> [`../BRIDGE_DEPLOYMENT_RUNBOOK.md`](../BRIDGE_DEPLOYMENT_RUNBOOK.md). +> These runbooks cover **running-system incidents** instead. + +--- + +## How to use these runbooks + +1. Identify the incident from the alert / symptom and open the matching runbook. +2. Work top-to-bottom. Every runbook is structured the same way: + - **Symptoms** β€” what you (or the alert) see. + - **Severity** β€” starting severity; escalate per the table below. + - **Diagnose** β€” commands and queries to confirm the root cause. + - **Mitigate** β€” fastest safe action to stop the bleeding. + - **Recover** β€” return to steady state. + - **Verify** β€” confirm the incident is resolved. + - **Post-incident** β€” follow-ups and prevention. +3. If two runbooks seem to apply, start with the one matching the *earliest* + symptom in the request path (e.g. provider outage before queue backlog). + +--- + +## Incident catalogue (top 10) + +| # | Runbook | Trigger / alert | Sev | +|---|---------|-----------------|-----| +| 01 | [Mobile money provider down](./01-provider-down.md) | `provider_circuit_breaker_state=1`, payout failures | P2 | +| 02 | [Database index bloat & slow queries](./02-database-index-bloat.md) | Rising query latency, `slow_query` logs | P3 | +| 03 | [High API latency](./03-high-api-latency.md) | P99 `http_request_duration_seconds` breach | P2 | +| 04 | [Queue backlog](./04-queue-backlog.md) | `total_depth` high, KEDA at max replicas | P2 | +| 05 | [Redis outage / failover](./05-redis-outage.md) | `/ready` shows `redis: down`, session/cache errors | P1 | +| 06 | [Stellar Horizon degradation](./06-stellar-horizon-degraded.md) | `horizon_node_health=0`, `horizon_node_failures_total` rising | P2 | +| 07 | [Database connection pool exhaustion](./07-db-pool-exhaustion.md) | `too many clients`, timeouts on `/ready` DB check | P1 | +| 08 | [Read-replica lag](./08-replica-lag.md) | `db_replica_lag_seconds` high, stale reads | P3 | +| 09 | [Ledger imbalance](./09-ledger-imbalance.md) | `reconcile:ledger` reports debits β‰  credits | P1 | +| 10 | [Elevated error rate](./10-elevated-error-rate.md) | Error-rate alert, `transaction_errors_total` spike | P2 | + +--- + +## Severity levels + +| Sev | Definition | Response time | Who | +|-----|------------|---------------|-----| +| **P1** | Funds at risk, or core deposit/withdraw path fully down | Immediate, page on-call | On-call + eng lead | +| **P2** | Major degradation, one provider/path affected, no data loss | < 15 min | On-call | +| **P3** | Minor degradation, elevated latency, capacity risk | < 1 hour (business hrs) | On-call / owner | +| **P4** | Cosmetic / no user impact | Next business day | Owner | + +Escalate a level whenever: funds could be lost, the incident lasts > 30 min +without mitigation, or a second subsystem starts failing. + +--- + +## Shared quick reference + +### Health & metrics endpoints + +| Endpoint | Purpose | +|----------|---------| +| `GET /health` | Liveness β€” process is up (returns `gitHash`). | +| `GET /ready` | Readiness β€” checks DB + Redis + shutdown state; 503 if any down. | +| `GET /health/lb` | Load-balancer check (DB, Redis, memory < 1 GB); 5 s cached. | +| `GET /health/queue` | BullMQ queue health summary. | +| `GET /health/queue/depth` | `total_depth` β€” the value KEDA scales workers on. | +| `GET /metrics` | Prometheus scrape (all app metrics). | +| `GET /metrics/queue_depth` | Prometheus queue-depth metrics. | +| `/admin/queues` | Bull-Board dashboard (inspect/retry jobs). | + +```bash +# Fast triage β€” is the app healthy end to end? +curl -s localhost:3000/ready | jq +curl -s localhost:3000/health/queue/depth | jq +``` + +### Key Prometheus metrics (see [`../metrics.md`](../metrics.md)) + +| Metric | Use | +|--------|-----| +| `http_request_duration_seconds` | API latency (histogram; P95/P99). | +| `http_requests_total` | Request volume & status codes. | +| `transaction_total{status}` | Deposit/payout throughput & success. | +| `transaction_errors_total{error_type}` | Transaction failures by cause. | +| `provider_circuit_breaker_state` | 0=closed, 1=open, 2=half-open. | +| `provider_failover_total` | Provider failover events. | +| `provider_response_time_seconds` | Per-provider latency. | +| `horizon_node_health` | Stellar Horizon node up/down. | +| `db_replica_lag_seconds` | Read-replica lag. | +| `cache_hit_ratio` | Redis cache effectiveness. | +| queue depth (via `/health/queue/depth`) | BullMQ backlog. | + +### Common tools + +```bash +# Ledger integrity check +npm run reconcile:ledger +npm run reconcile:ledger -- --date=2026-07-30 + +# Database index maintenance +npm run audit:indexes # find unused/bloated indexes +npm run reindex:bloated-indexes # REINDEX CONCURRENTLY eligible indexes + +# Migrations +npm run migrate:status +npm run migrate:up + +# Backups +npm run backup:create +npm run backup:verify + +# Admin CLI +npm run momo-cli -- --help +``` + +### Observability + +- **Grafana / Loki** β€” LogQL error-rate & latency queries in + [`../observability.md`](../observability.md). +- **Metrics reference** β€” [`../metrics.md`](../metrics.md). +- **Alerting** β€” PagerDuty (see `scripts/setup-pagerduty.sh`). +- **Log levels** β€” structured JSON; watch `ERROR`, `SECURITY`, `AUDIT`. + +--- + +## Golden rules + +1. **Communicate first.** Post in the incident channel before deep-diving. +2. **Mitigate before you diagnose** for P1/P2 β€” stop user impact, then find root cause. +3. **Never guess with funds.** For anything touching balances or the ledger, + halt the affected flow and reconcile before resuming. +4. **Write it down.** Capture a timeline as you go; it becomes the post-mortem.