feat: wrap all external dependencies with timeout/retry/circuit-breaker

[MerlinTheWhiz]

Description

Closes #97

Guards every external dependency behind consistent timeout, retry, and circuit-breaker policies. Adds structured error translation so clients receive safe, actionable HTTP responses. Propagates correlation IDs across the entire request lifecycle.

Changes

Resilience primitives (src/lib/resilience/):

• withTimeout(promise, ms, label) — races against a deadline with optional external abort signal
• withRetry(fn, opts) — bounded exponential backoff with full jitter, gated by an idempotency flag (non-idempotent writes never retry)
• createCircuitBreaker(name, opts) — in-memory state machine (CLOSED → OPEN → HALF_OPEN → CLOSED) with configurable thresholds and onStateChange callback for metrics
• DependencyError — structured error carrying dependency, action, retryable, statusCode, and userMessage

Adapter wrappers (each wraps its external call with the primitives above):

• MongoDB — CB in src/lib/mongodb.js
• Stellar Horizon — CB in src/lib/stellar/horizonClient.js
• Stellar RPC — shared client with timeout/retry/CB in src/lib/stellar/rpcClient.js, wired into entitlement.js, stellarIndexer.js, and ready/route.js
• Pinata — CB + callPinata() wrapper in src/lib/pinata.js; 7 call sites migrated across upload routes and ready route
• Nodemailer — CB + withEmailResilience() in src/lib/email.js
• Redis — CB + timeout on all operations in src/lib/cache/redis.js
• Webhooks — CB + traceparent propagation in src/lib/webhooks/dispatcher.js

Error translation (src/lib/api/hardening.js):

• Catches DependencyError and returns HTTP 503 with { error, dependency } body
• Increments circuit_breaker_open_total counter on each 503 response

Metrics (all adapters):

• circuit_breaker_state gauge set on every state transition
• circuit_breaker_open_total counter incremented on OPEN transitions
• rpc_errors_total counter on CB-open fast-fail

Correlation propagation:

• traceparent header on RPC and webhook requests
• x-correlation-id header on error responses via hardening layer

Fix: Removed dead code and duplicate module content in src/lib/entitlement.js that prevented the file from loading.

Testing

63 integration tests across 8 files covering:

• Primitive unit tests (timeout, retry, circuit breaker)
• Adapter-level fault injection (timeout, CB open, error toggling)
• Hardening layer DependencyError → HTTP mapping
• All tests pass

Checklist

• Every outbound request has a documented timeout and retry policy
• Non-idempotent writes are never blindly retried (idempotency guard default-off)
• Dependency degradation does not consume unbounded connections
• Fault-injection tests cover timeout, connection reset, and 5xx
• Metrics identify the failing dependency and circuit state
• User-facing errors give a safe retry/recovery action