feat: implement distributed lock timeouts, synthetic monitoring, prov…

[feyisaralawal]

…ider performance metrics, and distributed tracing

Resolves #250, #259, #258, #257
Closes #250
Closes #259
Closes #258
Closes #257

Detailed Explanation of Changes & Technical Implementation:

1. Distributed Transaction Lock with Timeout (Closes Implement Distributed Transaction Lock with Timeout #250)

• Enhanced LockManager in src/utils/lock.ts to support AcquireOptions (configurable ttl, timeoutMs, retryCount, retryDelay, exponential backoffFactor).
• Implemented LockAcquisitionTimeoutError to prevent processes from blocking indefinitely during high contention.
• Integrated Promise.race to enforce timeout constraints per lock operation.
• Added Prometheus metrics in src/utils/metrics.ts (lock_acquisition_total, lock_contention_total, lock_acquisition_duration_seconds) to track contention and acquisition latencies.

2. Synthetic Monitoring for Critical Flows (Closes Implement Synthetic Monitoring for Critical Flows #259)

• Built src/services/syntheticMonitoringService.ts to execute synthetic deposit, withdraw, and dispute transactions.
• Used SYNTHETIC_ prefixed references and isolated sandbox data to ensure zero impact on production data.
• Created src/jobs/syntheticMonitoringJob.ts and registered it in src/jobs/scheduler.ts to execute every minute.
• Exposed Prometheus metrics (synthetic_test_total, synthetic_test_duration_seconds, synthetic_test_success, synthetic_consecutive_failures) and integrated high-priority PagerDuty alerting within 2 minutes of consecutive flow failures.

3. Custom Metrics for Provider-Specific Performance (Closes Add Custom Metrics for Provider-Specific Performance #258)

• Developed src/services/providerMetricsService.ts to track 5-minute sliding window success rates, request counts, and execution latencies for mobile money providers (MTN, Airtel, Orange).
• Enhanced circuit breaker execution in src/utils/circuitBreaker.ts to record provider metrics and emit alerts on provider degradation (<85% success rate).
• Provisioned Grafana dashboard configuration (config/grafana/dashboards/provider_performance_comparison.json) comparing provider success rates, circuit breaker states, P95 latencies, and degradation alerts.

4. Distributed Tracing for Cross-Service Requests (Closes Implement Distributed Tracing for Cross-Service Requests #257)

• Enhanced src/tracer.ts with configurable sampling rates (TRACE_SAMPLE_RATE), context injection/extraction, and active trace metrics.
• Created Express distributed tracing middleware src/middleware/tracing.ts to extract trace headers (x-trace-id, traceparent, x-datadog-trace-id) and propagate context downstream.
• Instrument ingest-go/main.go to parse trace headers from incoming callback requests, tag Redis Stream messages with trace_id, and propagate trace IDs in HTTP responses.

Description

Brief description of changes.

Related Issue

Fixes #(issue number)

Type of Change

• Bug fix
• New feature
• Documentation update
• Code refactoring
• Performance improvement

Changes Made

Testing

How did you test these changes?

Checklist

• Code follows project style
• Self-reviewed my code
• Commented complex code
• Updated documentation
• No new warnings
• Added tests (if applicable)

Screenshots (if applicable)

Additional Notes

[drips-wave]

@feyisaralawal Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits