[OBS-01] OpenTelemetry + Structured Logging + Metrics (Full Stack)

[grantfox-oss]

Problem

Zero observability in production. No distributed tracing, no structured metrics, no centralized logging correlation.

Current State

Component	Observability
micopay/backend	pino JSON logs only — no trace IDs propagated, no metrics export
apps/api	Same — pino + basic helmet, no OTel
micopay/frontend	console.log — no error tracking, no web vitals
Soroban contracts	Event logs only — no tracing integration
External deps (Stellar RPC, Etherfuse, Didit)	No visibility into latency/errors/timeouts

Impact

• Debugging production issues = guesswork (no trace across services)
• No alerting on error rates, latency p99, dependency failures
• No capacity planning (no request rate, queue depth, DB pool metrics)
• Incident response time = hours instead of minutes

---

Solution: OpenTelemetry + Structured Logging

1. Backend Instrumentation (micopay/backend + apps/api)

// otel.ts — initialize once at app start
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION, ATTR_DEPLOYMENT_ENVIRONMENT } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'micopay-backend',
    [ATTR_SERVICE_VERSION]: process.env.APP_VERSION ?? 'dev',
    [ATTR_DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV ?? 'development',
  }),
  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({ url: process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT }),
    exportIntervalMillis: 10000,
  }),
  instrumentations: [getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-fastify': { enabled: true },
    '@opentelemetry/instrumentation-pg': { enabled: true },
    '@opentelemetry/instrumentation-fetch': { enabled: true },
    '@opentelemetry/instrumentation-http': { enabled: true },
  })],
});

sdk.start();

2. Custom Spans for Business Operations

// In trade.service.ts, stellar.service.ts, etc.
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('micopay.trade');

export async function createTrade(...) {
  return tracer.startActiveSpan('trade.create', async (span) => {
    try {
      span.setAttribute('trade.amount_mxn', amountMxn);
      span.setAttribute('trade.buyer_id', buyerId);
      span.setAttribute('trade.seller_id', sellerId);
      const result = await doCreateTrade(...);
      span.setAttribute('trade.id', result.id);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

3. Structured Logging with Trace Correlation

// logger.ts — pino + OTel
import pino from 'pino';
import { trace, context } from '@opentelemetry/api';

const logger = pino({
  formatters: {
    log(object) {
      const span = trace.getSpan(context.active());
      if (span) {
        const ctx = span.spanContext();
        object.trace_id = ctx.traceId;
        object.span_id = ctx.spanId;
      }
      return object;
    },
  },
});

4. Frontend (micopay/frontend) — Web Vitals + Error Tracking

// otel-web.ts
import { WebTracerProvider } from '@opentelemetry/sdk-trace-web';
import { ZoneContextManager } from '@opentelemetry/context-zone';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { DocumentLoadInstrumentation } from '@opentelemetry/instrumentation-document-load';
import { UserInteractionInstrumentation } from '@opentelemetry/instrumentation-user-interaction';
import { XMLHttpRequestInstrumentation } from '@opentelemetry/instrumentation-xml-http-request';
import { FetchInstrumentation } from '@opentelemetry/instrumentation-fetch';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const provider = new WebTracerProvider();
provider.addSpanProcessor(new BatchSpanProcessor(new OTLPTraceExporter()));
provider.register({ contextManager: new ZoneContextManager() });

registerInstrumentations({
  instrumentations: [
    new DocumentLoadInstrumentation(),
    new UserInteractionInstrumentation(),
    new XMLHttpRequestInstrumentation(),
    new FetchInstrumentation({ propagateTraceHeaderCorsUrls: [/.*/] }),
  ],
});

// Error boundary integration
window.addEventListener('error', (e) => {
  const span = trace.getTracer('micopay.frontend').startSpan('frontend.error');
  span.recordException(e.error);
  span.end();
});

5. Metrics to Export (Minimum Viable)

Metric	Type	Labels
http.server.request.duration	Histogram	method, route, status_code
http.server.request.active	UpDownCounter	method, route
db.query.duration	Histogram	query_type, table
stellar.rpc.call.duration	Histogram	method, network
etherfuse.api.call.duration	Histogram	endpoint, status
didit.verification.duration	Histogram	workflow_id, status
trade.created.total	Counter	status (success/failed)
kyc.gate.decision	Counter	operation, level, decision
queue.depth	Gauge	queue_name
cache.hit_ratio	Gauge	cache_name

---

Acceptance Criteria

• OTel SDK initialized in both micopay/backend and apps/api at startup
• Trace context propagated across: HTTP → Fastify → PG → Stellar RPC → Etherfuse → Didit
• Custom spans on: trade create/accept/release/refund, KYC gate, ramp orders, compliance jobs
• Structured logs include trace_id, span_id — queryable in Loki/Grafana
• Metrics exported to OTLP endpoint (Prometheus/Grafana Cloud/Datadog)
• Frontend exports: page load, interactions, fetch/XHR, errors
• Dashboard created: RED metrics (Rate, Errors, Duration) per service + business KPIs
• Alerts: error rate > 1%, p99 latency > 2s, Stellar RPC failures > 5/min, queue depth > 100
• Runbook: docs/OBSERVABILITY_RUNBOOK.md — how to trace a failed trade, correlate logs/metrics/traces

---

Configuration (Environment Variables)

# Backend
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318/v1/traces
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4318/v1/metrics
OTEL_SERVICE_NAME=micopay-backend
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.2.3

# Frontend (build-time)
VITE_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otel.example.com/v1/traces
VITE_OTEL_SERVICE_NAME=micopay-frontend

---

Files to Create/Modify

File	Action
micopay/backend/src/otel.ts	New — OTel SDK init
micopay/backend/src/logger.ts	New — pino + trace correlation
micopay/backend/src/services/*.ts	Add custom spans
apps/api/src/otel.ts	New — OTel SDK init
apps/api/src/logger.ts	New — pino + trace correlation
micopay/frontend/src/otel-web.ts	New — Web OTel init
micopay/frontend/src/main.tsx	Init OTel before app mount
.github/workflows/otel-collector.yml	New — deploy OTel collector (dev/staging)
docs/OBSERVABILITY_RUNBOOK.md	New — incident response guide
docker-compose.yml	Add otel-collector service

---

Labels

observability, complexity: high, GrantFox OSS, backend, frontend, opentelemetry, tracing, metrics, logging, monitoring

[Phantomcall]

I'd like to take this on. I have experience implementing OpenTelemetry across Node.js backends and frontend applications, including distributed tracing, structured logging, metrics, and production observability for API-driven systems.

For this issue, I'll introduce OpenTelemetry initialization for both micopay/backend and apps/api, enabling automatic instrumentation for Fastify, HTTP, PostgreSQL, and outbound requests while preserving the existing architecture. I'll add trace-aware structured logging by correlating trace_id and span_id with the existing Pino logger, ensuring logs, metrics, and traces can be followed end-to-end across backend services and external dependencies.

I'll instrument the core business flows with custom spans, including trade creation/acceptance/release/refund, KYC decisions, ramp orders, compliance jobs, Stellar RPC interactions, Etherfuse, and Didit integrations. I'll also expose the required OTLP metrics, document the exported metrics and labels, and add configuration for the required environment variables. On the frontend, I'll integrate the OpenTelemetry Web SDK to capture page loads, user interactions, fetch/XHR requests, and unhandled errors with trace propagation to the backend where supported.

In addition, I'll provide deployment support for a development/staging OpenTelemetry Collector, create an observability runbook describing how to investigate failed trades by correlating traces, logs, and metrics, and document the required dashboards and alerting recommendations. I'll ensure the implementation is configurable, secure, and degrades gracefully when observability endpoints are unavailable.

Before submitting the PR, I'll verify linting, tests, and builds across the affected packages, confirm trace propagation works end-to-end, and validate that sensitive information is never included in logs, spans, or metric labels.

GitHub: @PHANTOM_CALL