feat: weekly SLO error-budget report #543

.github/workflows/slo-weekly-report.yml

@@ -0,0 +1,30 @@
+name: SLO Weekly Report
+
+on:
+  schedule:
+    # Every Monday at 09:00 UTC
+    - cron: "0 9 * * 1"
+  workflow_dispatch:
+
+jobs:
+  slo-report:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "24"
+          cache: "npm"
+          cache-dependency-path: package-lock.json
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Run SLO weekly report
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+          PROMETHEUS_URL: ${{ secrets.PROMETHEUS_URL }}
+          BURN_RATE_DASHBOARD_URL: ${{ secrets.BURN_RATE_DASHBOARD_URL }}
+        run: npx tsx scripts/slo-weekly-report.ts

docs/observability.md

@@ -20,6 +20,10 @@ ChronoPay uses `X-Request-Id` for end-to-end request correlation.
 
 ## Service Level Objectives (SLOs)
 
+Route-level objectives and burn-rate metrics are defined in `src/metrics/sloMetrics.ts` (`booking_intent`, `slots_list`, `checkout`, `escrow_listener`).
+
+A **weekly error-budget report** is posted to Slack every Monday (see [slo-weekly-report.md](./slo-weekly-report.md) and `ops/dashboards/slo-burn-rate.json`).
+
 ### Booking Intents
 - **Availability & Latency:** 99.9% of booking-intent creates (`POST /api/v1/booking-intents`) complete in under 500ms over a rolling 30-day window.
 

docs/slo-weekly-report.md

@@ -0,0 +1,73 @@
+# Weekly SLO Error-Budget Report
+
+Automated weekly Slack summary of SLO error-budget consumption per route.
+
+## Routes and objectives
+
+Aligned with `src/metrics/sloMetrics.ts`:
+
+| Route | Objective |
+|-------|-----------|
+| `booking_intent` | 99.9% |
+| `slots_list` | 99.5% |
+| `checkout` | 99.99% |
+| `escrow_listener` | 99% |
+
+## Budget math
+
+For each route over the past 7 days:
+
+```
+errorBudget      = 1 − sloObjective
+observedErrorRate = badEvents / totalRequests   (0 when no traffic)
+consumedFraction  = observedErrorRate / errorBudget
+remainingBudget   = clamp(1 − consumedFraction, 0, 1)
+```
+
+Weekly consumption shown in Slack is `consumedFraction × 100%` of the error budget.
+
+## Data sources
+
+The script (`scripts/slo-weekly-report.ts`) resolves data in order:
+
+1. **Prometheus counters** — `increase(slo_requests_total[7d])` and `increase(slo_bad_events_total[7d])` (or `slo_route_*` variants) when present.
+2. **`slo_burn_rate` gauge** — max over the week per route when counters are absent.
+3. **No-traffic defaults** — all four routes at 100% remaining with a _no traffic_ marker when Prometheus is empty or unreachable.
+
+## Slack message
+
+- Per-route leaderboard sorted worst-first (lowest remaining budget)
+- Remaining budget %, weekly consumption %, status (Exhausted / Critical / At Risk / Healthy)
+- Link to the [SLO burn-rate dashboard](../ops/dashboards/slo-burn-rate.json) (`uid: slo-burn-rate`)
+
+## Running locally
+
+```bash
+SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... \
+PROMETHEUS_URL=http://localhost:9090 \
+BURN_RATE_DASHBOARD_URL=https://grafana.example.com/d/slo-burn-rate \
+npx tsx scripts/slo-weekly-report.ts
+```
+
+Environment defaults (when unset):
+
+- `PROMETHEUS_URL` — empty (posts no-traffic defaults)
+- `BURN_RATE_DASHBOARD_URL` — `https://grafana.example.com/d/slo-burn-rate`
+
+## CI schedule
+
+GitHub Actions workflow `.github/workflows/slo-weekly-report.yml` runs every **Monday at 09:00 UTC** and supports manual `workflow_dispatch`.
+
+Required secrets:
+
+- `SLACK_WEBHOOK_URL` (required)
+- `PROMETHEUS_URL` (optional)
+- `BURN_RATE_DASHBOARD_URL` (optional)
+
+Slack delivery failures throw so the cron job fails loudly.
+
+## Tests
+
+```bash
+npm test -- --testPathPattern='slo-weekly-report' --coverage=false
+```

ops/dashboards/slo-burn-rate.json

@@ -0,0 +1,186 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "datasource",
+          "uid": "grafana"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "slo_burn_rate{window=\"5m\"}",
+          "legendFormat": "{{route}} (5m)",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "SLO Burn Rate — 5m window",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 8
+      },
+      "id": 2,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "slo_burn_rate{window=\"1h\"}",
+          "legendFormat": "{{route}} (1h)",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "SLO Burn Rate — 1h window",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 16
+      },
+      "id": 3,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "slo_burn_rate{window=\"6h\"}",
+          "legendFormat": "{{route}} (6h)",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "SLO Burn Rate — 6h window",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 24
+      },
+      "id": 4,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "max by (route) (slo_burn_rate)",
+          "legendFormat": "{{route}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Max Burn Rate by Route",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 0.5 },
+              { "color": "red", "value": 1 }
+            ]
+          }
+        }
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 24
+      },
+      "id": 5,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "clamp_max(1 - max by (route) (max_over_time(slo_burn_rate[7d])), 1)",
+          "legendFormat": "{{route}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Estimated Weekly Budget Remaining",
+      "type": "stat"
+    }
+  ],
+  "title": "SLO Burn Rate",
+  "uid": "slo-burn-rate",
+  "version": 1
+}