Add Prometheus Metrics Endpoint for Observability Across All Services

[therealjhay]

Category: New Features & Improvements
Difficulty: High

Description:
None of the four services expose Prometheus-compatible metrics. There are no:

• HTTP request counters, duration histograms, or in-flight request gauges
• Database query duration and error counters
• Redis connection health metrics
• BullMQ queue depth, job duration, and failure counters
• Business-level metrics (payments created, settlements processed, events indexed)
• Rate-limit hit counters

Without observability, it is impossible to monitor production health, detect performance regressions, or set up alerting for anomalies. A shared @fastify/metrics or prom-client setup should be added to all services.

Location:
All four services (no metrics implementation exists)

Acceptance Criteria:

• Add the @fastify/metrics plugin to all four services (or a custom /metrics endpoint using prom-client)
• Export default Prometheus metrics (process CPU, memory, event loop lag, garbage collection)
• Add custom business metrics: bettapay_payments_total, bettapay_settlements_total, bettapay_events_indexed_total
• Add BullMQ queue metrics: bullmq_queue_depth, bullmq_job_duration_seconds
• Add a GET /metrics endpoint (protected by serviceAuth) exposing all metrics in Prometheus text format
• Create a Grafana dashboard JSON (as docs/grafana-dashboard.json) with panels for each service
• Write a test that verifies the /metrics endpoint returns valid Prometheus-formatted output

Technical Notes:

• @fastify/metrics automatically registers HTTP metrics. Pair it with prom-client for custom metrics.
• BullMQ v5+ has <queue>.getJobCounts() for queue depth; wrap in a Gauge collector.

[grantfox-oss]

🦊 GrantFox — @Ebenezer199914 has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #255)
2. Your PR will be reviewed by the Betta-Pay maintainers

Good luck! Track your progress on GrantFox.