Skip to content

Commit ad42a42

Browse files
Chucks1093Chucks1093
authored
Merge pull request #575 from Ajibose/feat/health-check-docs
docs(health): document all health endpoint response fields and states
2 parents 6e32f7a + 9a0c809 commit ad42a42

4 files changed

Lines changed: 924 additions & 36 deletions

File tree

‎docs/health-endpoints.md‎

Lines changed: 270 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,55 @@
11
# Health and Readiness Endpoints
22

3-
The health module exposes three endpoints, each with a different contract.
3+
The health module exposes four endpoints with different contracts and intended consumers.
4+
5+
## Liveness vs Readiness
6+
7+
| Endpoint | Purpose | Consumer |
8+
| ----------------------------- | ------------------------------------------------------------ | ----------------------------------------- |
9+
| `GET /api/v1/health` | Liveness — confirms the process is alive | Load balancers, uptime monitors |
10+
| `GET /api/v1/health/ready` | Readiness — confirms all critical dependencies are reachable | Kubernetes `readinessProbe`, deploy gates |
11+
| `GET /api/v1/health/detailed` | Diagnostics — full system snapshot | Operators, dashboards |
12+
| `GET /api/v1/health/indexer` | Worker heartbeat — confirms the indexer is running | Alerting, internal monitoring |
13+
14+
**Liveness** never probes dependencies. It responds `200` as long as the event loop is processing requests. Use it wherever a dependency failure should not pull the instance from rotation (e.g., when a downstream DB blip should not make the server appear down).
15+
16+
**Readiness** actively pings each critical dependency. A single `fail` result flips the response to `503`, signalling to the orchestrator that this instance should stop receiving traffic until the dependency recovers.
17+
18+
---
419

520
## `GET /api/v1/health` — liveness
621

7-
A minimal "the process is up" check. Always `200` while the event loop is
8-
healthy. Safe for load balancers and uptime monitors that should not fan out
9-
to dependencies.
22+
A minimal "the process is up" check. Always `200` while the event loop is healthy. No dependency probing.
23+
24+
### Response shape
25+
26+
```json
27+
{
28+
"success": true,
29+
"message": "OK",
30+
"timestamp": "2026-04-28T16:00:00.000Z"
31+
}
32+
```
33+
34+
### Fields
35+
36+
| Field | Type | Description |
37+
| ----------- | ----------------- | -------------------------------------------------- |
38+
| `success` | boolean | Always `true`. |
39+
| `message` | string | Human-readable confirmation string. Always `"OK"`. |
40+
| `timestamp` | string (ISO-8601) | When the response was built. |
41+
42+
### HTTP status codes
43+
44+
| Code | Condition |
45+
| ----- | ---------------------------------------- |
46+
| `200` | Always — liveness never returns non-200. |
47+
48+
---
1049

1150
## `GET /api/v1/health/ready` — readiness
1251

13-
Probes critical dependencies (database, cache config). Returns `200` when
14-
every probe passes and `503` otherwise.
52+
Probes critical dependencies (database, cache config). Returns `200` when every probe passes and `503` otherwise.
1553

1654
### Response shape
1755

@@ -29,27 +67,172 @@ every probe passes and `503` otherwise.
2967

3068
### Fields
3169

32-
| Field | Type | Notes |
33-
| ----------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
34-
| `ready` | boolean | `true` only when every check is `ok`. Maps directly to the HTTP status (`200` vs `503`). |
35-
| `timestamp` | string (ISO-8601) | When the response was built. |
36-
| `latencyMs` | number | **Total** wall-clock duration of the readiness probe — sum of every check plus orchestration overhead. Useful for dashboards and SLO tracking. |
37-
| `checks` | array | Per-dependency results, each with its own `name`, `status`, optional `latencyMs`, and optional `error`. |
70+
| Field | Type | Description |
71+
| ----------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------- |
72+
| `ready` | boolean | `true` only when every check returns `"ok"`. **This field determines the HTTP status** — `true` → `200`, `false` → `503`. |
73+
| `timestamp` | string (ISO-8601) | When the response was built. |
74+
| `latencyMs` | number | Total wall-clock duration of the entire readiness probe in milliseconds. Useful for dashboards and SLO tracking. |
75+
| `checks` | array | Per-dependency probe results. See the checks table below. |
76+
77+
#### `checks` array entry fields
78+
79+
| Field | Type | Present when | Description |
80+
| ----------- | ------ | ------------------------------------------- | -------------------------------------------------------------------------------------------------- |
81+
| `name` | string | Always | Dependency name. Current values: `"database"`, `"cache"`. |
82+
| `status` | string | Always | `"ok"` — probe passed; `"fail"` — probe failed. |
83+
| `latencyMs` | number | `status: "ok"` and check measured a latency | Round-trip time for this probe in milliseconds. |
84+
| `error` | string | `status: "fail"` | Human-readable reason for the failure. No internal hostnames, connection strings, or stack traces. |
3885

39-
The payload is intentionally public-safe: no internal hostnames,
40-
connection strings, or stack traces are included even when a check fails.
86+
#### Probe descriptions
87+
88+
| Probe name | What is checked | Healthy value | Unhealthy value |
89+
| ---------- | --------------------------------------------------------------------- | ---------------------------- | -------------------------- |
90+
| `database` | Issues `SELECT 1` against the primary database. | `"ok"` + `latencyMs` present | `"fail"` + `error` present |
91+
| `cache` | Verifies that the HTTP cache layer config constant is a valid number. | `"ok"` | `"fail"` + `error` present |
92+
93+
### Non-200 trigger
94+
95+
The `ready` field is the sole trigger. When **any** check in the `checks` array has `status: "fail"`, `ready` is `false` and the HTTP status is `503`.
96+
97+
The payload is intentionally public-safe: no internal hostnames, connection strings, or stack traces are included even when a check fails.
98+
99+
---
41100

42101
## `GET /api/v1/health/detailed` — diagnostics
43102

44-
Full system snapshot including memory, uptime, system info, database
45-
response time, chain-sync lag and per-service health flags. Intended for
46-
operators rather than load balancers — cheaper liveness/readiness paths
47-
should be preferred for automated probing.
103+
Full system snapshot including memory, uptime, system info, database response time, chain-sync lag, and per-service health flags. Intended for operators and dashboards — use the cheaper liveness and readiness paths for automated probing.
104+
105+
### Response shape
106+
107+
```json
108+
{
109+
"success": true,
110+
"message": "Access Layer server is running",
111+
"timestamp": "2026-04-28T16:00:00.000Z",
112+
"version": "1.0.0",
113+
"environment": "production",
114+
"uptime": 3600.5,
115+
"memory": {
116+
"used": 48.32,
117+
"total": 64.0
118+
},
119+
"system": {
120+
"platform": "linux",
121+
"nodeVersion": "v20.11.0"
122+
},
123+
"timeouts": {
124+
"database_timeout_ms": 5000,
125+
"cache_timeout_ms": 300000
126+
},
127+
"database": {
128+
"status": "connected",
129+
"responseTime": 4
130+
},
131+
"syncing": {
132+
"status": "in-sync",
133+
"latestIndexedLedger": 12345,
134+
"observedHeadLedger": 12400,
135+
"syncLagLedgers": 55
136+
},
137+
"services": [
138+
{ "name": "API Server", "status": "healthy" },
139+
{ "name": "Database", "status": "healthy" },
140+
{ "name": "Chain Sync", "status": "healthy" }
141+
]
142+
}
143+
```
144+
145+
### Top-level fields
146+
147+
| Field | Type | Description |
148+
| ------------- | ----------------- | --------------------------------------------------------------------------------------------------- |
149+
| `success` | boolean | Always `true` in a normal response. `false` only on an unexpected exception. |
150+
| `message` | string | Human-readable status string. `"Access Layer server is running"` when healthy. |
151+
| `timestamp` | string (ISO-8601) | When the response was built. |
152+
| `version` | string | Application version string. |
153+
| `environment` | string | Deployment environment. Mirrors `MODE` env var (e.g. `"production"`, `"staging"`, `"development"`). |
154+
| `uptime` | number (seconds) | Seconds since the Node.js process started (`process.uptime()`). Always ≥ 0. |
155+
| `memory` | object | JVM heap snapshot. See below. |
156+
| `system` | object | Runtime identity. See below. |
157+
| `timeouts` | object | Public-safe dependency timeout values. See below. |
158+
| `database` | object | Database connectivity status. See below. |
159+
| `syncing` | object \| absent | Chain indexer sync lag. Absent when the sync status check itself errors. |
160+
| `services` | array | Rolled-up health flags for each major component. See below. |
161+
162+
### `memory` fields
163+
164+
| Field | Type | Description |
165+
| ------- | ------ | ----------------------------------------------------------------------- |
166+
| `used` | number | Node.js heap memory currently used, in **megabytes** (rounded to 2 dp). |
167+
| `total` | number | Node.js heap memory allocated, in **megabytes** (rounded to 2 dp). |
168+
169+
### `system` fields
170+
171+
| Field | Type | Description |
172+
| ------------- | ------ | -------------------------------------------------------------- |
173+
| `platform` | string | Operating system platform string (e.g. `"linux"`, `"darwin"`). |
174+
| `nodeVersion` | string | Node.js version string (e.g. `"v20.11.0"`). |
48175

49-
### Field order and grouping
176+
### `timeouts` fields
50177

51-
The detailed health payload should preserve this top-level order to keep
52-
JSON snapshots predictable for contributors and dashboards:
178+
Public-safe dependency timeout configuration. No connection strings, hostnames, or credentials are included.
179+
180+
| Field | Type | Description |
181+
| --------------------- | ------ | -------------------------------------------------- |
182+
| `database_timeout_ms` | number | Configured database query timeout in milliseconds. |
183+
| `cache_timeout_ms` | number | Configured public HTTP cache TTL in milliseconds. |
184+
185+
### `database` fields
186+
187+
| Field | Type | Present when | Description |
188+
| -------------- | ------ | --------------------- | ---------------------------------------------------------------------- |
189+
| `status` | string | Always | `"connected"` — `SELECT 1` succeeded; `"disconnected"` — query failed. |
190+
| `responseTime` | number | `status: "connected"` | Round-trip time for the DB probe in milliseconds. |
191+
192+
#### Database states
193+
194+
| `status` | Meaning | HTTP impact |
195+
| ---------------- | ------------------------------------------- | ----------------------------------------------------- |
196+
| `"connected"` | Database is reachable and responding. | Healthy. |
197+
| `"disconnected"` | Database probe threw an error or timed out. | **503** in `production`; `200` in other environments. |
198+
199+
### `syncing` fields
200+
201+
Present when the chain indexer sync status is available. Absent if the sync check itself errors.
202+
203+
| Field | Type | Description |
204+
| --------------------- | ------ | ----------------------------------------------------------------------------------- |
205+
| `status` | string | `"in-sync"` — lag is within threshold; `"degraded"` — lag exceeds the threshold. |
206+
| `latestIndexedLedger` | number | Most recent ledger the indexer has processed. |
207+
| `observedHeadLedger` | number | Latest ledger observed on the chain. |
208+
| `syncLagLedgers` | number | Difference: `observedHeadLedger − latestIndexedLedger`. Threshold: **100 ledgers**. |
209+
210+
#### Sync states
211+
212+
| `status` | Condition | `syncLagLedgers` |
213+
| ------------ | ---------------------------------- | ---------------- |
214+
| `"in-sync"` | Lag is within the 100-ledger limit | 0 – 100 |
215+
| `"degraded"` | Lag exceeds 100 ledgers | > 100 |
216+
217+
The sync state does **not** affect the HTTP status code of this endpoint. Use `GET /api/v1/health/indexer` for a dedicated alerting-friendly probe.
218+
219+
### `services` array
220+
221+
A rolled-up health summary of each major component.
222+
223+
| `name` | `status: "healthy"` condition | `status: "unhealthy"` condition |
224+
| -------------- | ------------------------------------------------- | ------------------------------------- |
225+
| `"API Server"` | Always healthy (present means the server is up). | Never unhealthy in normal operation. |
226+
| `"Database"` | `database.status === "connected"`. | `database.status === "disconnected"`. |
227+
| `"Chain Sync"` | `syncing.status !== "degraded"` (or sync absent). | `syncing.status === "degraded"`. |
228+
229+
### Non-200 trigger
230+
231+
`database.status === "disconnected"` **and** `environment === "production"` triggers a `503`. In non-production environments the endpoint always returns `200` regardless of database connectivity.
232+
233+
### Field order
234+
235+
The top-level JSON keys are guaranteed to appear in this order:
53236

54237
1. `success`
55238
2. `message`
@@ -59,24 +242,75 @@ JSON snapshots predictable for contributors and dashboards:
59242
6. `uptime`
60243
7. `memory`
61244
8. `system`
62-
9. `database`
63-
10. `syncing`
64-
11. `services`
245+
9. `timeouts`
246+
10. `database`
247+
11. `syncing`
248+
12. `services`
249+
250+
---
251+
252+
## `GET /api/v1/health/indexer` — worker heartbeat
65253

66-
Nested grouping follows the same convention:
254+
Reports the liveness state of the indexer background worker. The indexer calls `POST /api/v1/health/indexer/heartbeat` after each successful run; this endpoint exposes the resulting state.
67255

68-
- `memory`: `used`, `total`
69-
- `system`: `platform`, `nodeVersion`
70-
- `database`: `status`, `responseTime` (when connected)
71-
- `syncing`: `status`, `latestIndexedLedger`, `observedHeadLedger`, `syncLagLedgers`
72-
- `services`: ordered as `API Server`, `Database`, `Chain Sync`
73-
- `timeouts`: `database_timeout_ms`, `cache_timeout_ms`
256+
### Response shape
257+
258+
```json
259+
{
260+
"success": true,
261+
"data": {
262+
"service": "indexer",
263+
"status": "healthy",
264+
"lastSuccessfulRun": "2026-04-28T15:59:00.000Z",
265+
"staleSinceMs": null
266+
}
267+
}
268+
```
269+
270+
### Fields
271+
272+
| Field | Type | Description |
273+
| ------------------------ | -------------- | ---------------------------------------------------------------------------- |
274+
| `success` | boolean | Always `true`. |
275+
| `data.service` | string | Always `"indexer"`. |
276+
| `data.status` | string | Current worker state. See status table below. |
277+
| `data.lastSuccessfulRun` | string \| null | ISO-8601 timestamp of the most recent heartbeat, or `null` if none recorded. |
278+
| `data.staleSinceMs` | number \| null | Milliseconds since the heartbeat became stale, or `null` when not stale. |
279+
280+
#### Worker status values
74281

75-
### Detailed health fields
282+
| `data.status` | Condition | `lastSuccessfulRun` | `staleSinceMs` | HTTP status |
283+
| ------------- | ------------------------------------------------------------------------------------- | ------------------- | --------------- | ----------- |
284+
| `"unknown"` | No heartbeat has ever been recorded (fresh deploy or restart). | `null` | `null` | `200` |
285+
| `"healthy"` | Last heartbeat is within the stale threshold. | ISO-8601 string | `null` | `200` |
286+
| `"degraded"` | Last heartbeat exceeded the stale threshold (`INDEXER_HEARTBEAT_STALE_THRESHOLD_MS`). | ISO-8601 string | positive number | `503` |
76287

77-
The `/api/v1/health/detailed` response also includes an explicit `timeouts` object with public-safe dependency timeout values:
288+
### Non-200 trigger
78289

79-
- `database_timeout_ms` (number) — configured database query timeout in milliseconds.
80-
- `cache_timeout_ms` (number) — configured public cache timeout in milliseconds.
290+
`data.status === "degraded"` is the sole trigger for `503`. Both `"unknown"` and `"healthy"` return `200`.
291+
292+
---
293+
294+
## `POST /api/v1/health/indexer/heartbeat` — record worker run
295+
296+
Called by the indexer worker after each successful run to reset the stale timer.
297+
298+
### Response shape
299+
300+
```json
301+
{
302+
"success": true,
303+
"data": {
304+
"recorded": true,
305+
"timestamp": "2026-04-28T16:00:00.000Z"
306+
},
307+
"message": "Heartbeat recorded"
308+
}
309+
```
81310

82-
These values are intentionally numeric-only and do not include connection strings, hostnames, credentials, or other internal topology details.
311+
| Field | Type | Description |
312+
| ---------------- | ----------------- | -------------------------------------------- |
313+
| `success` | boolean | Always `true`. |
314+
| `data.recorded` | boolean | Always `true` when the heartbeat was stored. |
315+
| `data.timestamp` | string (ISO-8601) | The time the heartbeat was recorded. |
316+
| `message` | string | Always `"Heartbeat recorded"`. |

‎src/modules/health/health.controllers.integration.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ function mockResponse(): Response & { statusCode: number; body: any } {
4343
res.body = payload;
4444
return res;
4545
};
46+
res.setHeader = () => res;
4647
return res;
4748
}
4849

‎src/modules/health/health.controllers.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ function mockResponse(): Response & { statusCode: number; body: any } {
6262
return res;
6363
};
6464

65+
res.setHeader = () => res;
66+
6567
return res;
6668
}
6769

0 commit comments

Comments
 (0)