feat: add request observability and Stellar reconciliation worker

[Godsmiracle001]

Closes #16
Closes #17

Summary

This PR adds two production-facing infrastructure improvements:

1. Request observability for the HTTP app
2. A background worker that reconciles stale pending Stellar state by reusing the existing payment verification path

What Changed

Observability

• Added request correlation IDs
• Accepts X-Request-Id when provided, otherwise generates a UUID
• Returns the resolved request ID in the X-Request-Id response header
• Includes the request ID in structured Winston JSON logs
• Added Prometheus-style GET /metrics
• Tracks low-cardinality HTTP metrics by method, normalized route template, and status class
• Enriched GET /health with process uptime and request ID
• Avoids logging request bodies and Authorization headers

Background reconciliation

• Added an env-flagged in-process worker for stale pending Stellar reconciliation
• Selects a bounded batch of stale pending investments / transactions with known Stellar hashes
• Reuses the existing verification logic instead of duplicating Horizon business rules
• Handles per-item errors without crashing the process
• Yields between items so a tick does not starve the event loop
• Supports graceful shutdown on SIGTERM / SIGINT

Config

Added:

• METRICS_ENABLED
• STELLAR_RECONCILIATION_ENABLED
• STELLAR_RECONCILIATION_INTERVAL_MS
• STELLAR_RECONCILIATION_BATCH_SIZE
• STELLAR_RECONCILIATION_GRACE_PERIOD_MS
• STELLAR_RECONCILIATION_MAX_RUNTIME_MS

Also wired .env loading in app bootstrap / DB config.

Prerequisites

Branch stacking strategy:

• This PR assumes the behavior from Bootstrap Express API: response envelope, errors, health, /api/v1 #5, Investment commitments: fractional funding and DB concurrency safety #6, Stellar Horizon payment verification and idempotent DB reconciliation #7, and Migration: link transactions to invoices and investments (audit FKs) #13
• If any of those are not merged yet, this PR should be reviewed as stacked on top of them
• The reconciliation worker specifically reuses the existing verification path from Stellar Horizon payment verification and idempotent DB reconciliation #7 and existing investment/transaction linkage from Migration: link transactions to invoices and investments (audit FKs) #13

Testing

Automated:

• Added tests for request ID generation and propagation
• Added tests for Prometheus metrics output
• Added tests for reconciliation success, partial failure, runtime budget cutoff, and scheduler stop behavior
• No live network calls in CI; Horizon interactions are mocked

pm test

pm run lint

pm run type-check

pm run build

Manual checklist:

• GET /health returns 200 and includes uptimeSeconds
• GET /metrics returns Prometheus text format
• Two concurrent requests produce different request IDs in logs
• Supplying X-Request-Id is reflected in logs and response headers
• With STELLAR_RECONCILIATION_ENABLED=true, the worker starts and shuts down cleanly with the process

Operational Notes

• Metrics label cardinality is intentionally low
• Current reconciliation worker assumes a single active replica
• Distributed leader election / advisory locking is out of scope for this PR

[drips-wave]

@Godsmiracle001 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits