diff --git a/docs/CHANGELOG-native-metrics.md b/docs/CHANGELOG-native-metrics.md new file mode 100644 index 0000000..f9664cb --- /dev/null +++ b/docs/CHANGELOG-native-metrics.md @@ -0,0 +1,92 @@ +# O que está subindo — Native Metrics + Auto-Instrumentation (lib-observability) + +> Registro vivo do que este trabalho adiciona à lib. Vira o corpo do PR único p/ develop. +> Branch: `feat/native-metrics-phase1` (base develop pós-#30). Atualizar a cada feature. +> Ref: pre-dev `docs/pre-dev/observability-metrics-standardization/` · contrato `docs/metrics-contract.md` + +## Objetivo +Padronizar a emissão de métricas transversais (a lib emite por default, nome/unidade/labels iguais p/ todo serviço) e oferecer helpers de auto-instrumentação de DB/cache/fila — para acabar com a bagunça de métricas por-app e aposentar spans manuais de infra (origem de ~27k séries de alta cardinalidade). + +## Não confundir +- **PR #30 (já mergeado)** = fix de span.name (route template). Pré-requisito, NÃO faz parte deste trabalho. +- **Adoção nas apps (midaz/ledger)** = outro repo, outro PR (Fase 3). Este PR é SÓ a lib. + +--- + +## FASE 1 — Métricas nativas (✅ implementada, testada) + +| Feature | Métrica | Tipo | Detalhe | +|---|---|---|---| +| T1 Contrato | — | doc | `docs/metrics-contract.md`: nomes/unidade(s)/buckets/labels permitidos×proibidos/no-op. Fundação. | +| T2 RED gRPC server | `rpc.server.duration` | Float64 Histogram (s) | hook no interceptor server existente. Labels: rpc.system=grpc, rpc.method, rpc.grpc.status_code, error.type (só !=OK), tenant.id | +| T2 RED gRPC client | `rpc.client.duration` | Float64 Histogram (s) | NOVO UnaryClientInterceptor (não existia). Labels iguais, sem tenant.id. Propaga trace. | +| T3 Runtime Go | `go.*` (memory/goroutine/gc) | contrib/runtime | opt-in via `TelemetryConfig.EnableRuntimeMetrics` (default-off). MinReadMemStats 15s. | +| T4 In-flight HTTP | `http.server.active_requests` | Int64 UpDownCounter ({request}) | inc antes / dec depois. Label http.request.method. Detecta saturação. | + +Arquivos: middleware/telemetry.go, tracing/otel.go, go.mod (+contrib/instrumentation/runtime v0.69.0), 3 test files (16 testes). Verificado: go test unit PASS, vet OK, lint 0, ManualReader (emissão real), zero label proibido, no-op safe. + +## DESACOPLAMENTO Fiber v3 ↔ core (✅ implementado, testado) — BREAKING + +Objetivo: tirar a dependência de `github.com/gofiber/fiber/v3` do núcleo da lib para que apps ainda em **Fiber v2** possam usar TUDO menos o middleware HTTP (core `NewTelemetry`, runtime metrics, messaging, gRPC, DB/cache). + +Causa raiz: `tracing/otel.go` importava `fiber/v3` só por 2 helpers HTTP. Isso contaminava o pacote `tracing` inteiro e, transitivamente, `messagingobs` (importa tracing), `runtime` e os interceptors gRPC — `go list -deps ./messagingobs` mostrava 13 deps de fiber. + +### O que mudou +- **`tracing` agora é fiber-free.** Removido o import de `fiber/v3` (e `redaction`/`observability`, que só existiam por causa das 2 funções movidas). +- **Novo pacote `grpcmiddleware/`** (fiber-free): interceptors gRPC saíram de `middleware` (que importa fiber). HTTP (`WithTelemetry`/`EndTracingSpans`) fica onde estava (fiber, correto). +- **Novo pacote `telemetrycore/`** (fiber-free): coletor de métricas de sistema (singleton único), compartilhado por HTTP e gRPC — evita duas goroutines de coleta quando a app usa os dois transportes. + +### Símbolos movidos (BREAKING — ajustar import path nos callers) +| Símbolo | Antes | Depois | +|---|---|---| +| `SetSpanAttributeForParam(c fiber.Ctx, ...)` | `tracing` (`.../v2/tracing`) | `middleware` (`.../v2/middleware`) | +| `ExtractHTTPContext(ctx, c fiber.Ctx)` | `tracing` | `middleware` | +| `WithTelemetryInterceptor` (gRPC) | `middleware.TelemetryMiddleware` | `grpcmiddleware.TelemetryMiddleware` | +| `EndTracingSpansInterceptor` (gRPC) | `middleware.TelemetryMiddleware` | `grpcmiddleware.TelemetryMiddleware` | +| `UnaryClientInterceptor` (gRPC) | `middleware.TelemetryMiddleware` | `grpcmiddleware.TelemetryMiddleware` | +| `ResolveTenantIDFromGRPC` | `middleware` (mantido lá tb.) | também em `grpcmiddleware` | +| `StopMetricsCollector` / `DefaultMetricsCollectionInterval` | `middleware` (mantidos p/ compat) | fonte agora em `telemetrycore` | + +Novo construtor gRPC: `grpcmiddleware.NewTelemetryMiddleware(tl)` (mesma assinatura de `middleware.NewTelemetryMiddleware`). + +### Fix externo pendente (midaz — outro repo, outro PR) +`midaz` `pkg/net/http/withBody.go:235` usa `SetSpanAttributeForParam` importando de `tracing`. Trocar para o pacote `middleware`: +`github.com/LerianStudio/lib-observability/v2/middleware.SetSpanAttributeForParam`. +(Apps que consomem os interceptors gRPC via `middleware` também precisam trocar para `grpcmiddleware`.) + +### Verificação (`go list -deps`, deps de gofiber) +| Pacote | Antes | Depois | +|---|---|---| +| `tracing` | 13 | **0** | +| `messagingobs` | 13 | **0** | +| `runtime` | 0 | 0 | +| `grpcmiddleware` (gRPC) | (era `middleware`=13) | **0** | +| `telemetrycore` | — | **0** | +| `middleware` (HTTP `WithTelemetry`) | 13 | 13 (correto, inalterado) | + +Comportamento 100% preservado (só MOVE código; nenhuma lógica de métrica/telemetria alterada). `go test -tags=unit ./...` PASS em todos os pacotes; golangci-lint (wsl_v5) 0 issues. Core OTel mantido em v1.44.0. + +## FASE 2 — Wrappers de auto-instrumentação (a implementar) + +| Feature | Métrica | Helper | Cobre | +|---|---|---|---| +| SQL | `db.client.operation.duration` (s) | `InstrumentSQLDB(*sql.DB)` (otelsql v0.43.0) | Postgres + MySQL/MariaDB (database/sql) | +| Cache | `db.client.operation.duration` (s) | `WrapRedis(client)` (redisotel v9.17.2) | Redis + Valkey (mesmo driver go-redis) | +| ~~Doc-DB (Mongo)~~ | — | — | **ADIADO** — otelmongo v2 sem release oficial (só pseudo-version que arrastaria otel core >v1.44.0); v1 deprecated. Entra quando v2 for tagueado. Ver BACKLOG. | +| Mensageria | `messaging.client.operation.duration` / `messaging.process.duration` (s) | InstrumentPublish/Consume (hand-roll s/ helpers de propagação existentes) | RabbitMQ (contrato compartilhado c/ lib-streaming p/ RedPanda) | + +Guardrails (todos): unidade s, sem query text/params/PII como label (contrato §PROIBIDO), no-op safe, instrumento 1×. +Boundary: lib expõe HELPER; conexão/dbresolver fica na app/lib-commons. Para SQL, aplicar em cada *sql.DB ANTES do dbresolver. + +## FORA deste PR +- Adoção no ledger + aposentar spans manuais (Fase 3, repo midaz). +- Dashboard genérico + rollout (Fase 4). +- Business metrics (skill Ring dev-sre, frente paralela). +- IBM MQ (backlog isolado). RedPanda/Kafka (lib-streaming, repo separado). + +## BACKLOG (fora deste PR, rastreado) +- **Mongo helper (`mongoobs`):** implementar quando `otelmongo v2` (`go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/v2/mongo/otelmongo`) tiver release oficial que resolva p/ otel core v1.44.x. Hoje: v2 sem tag (só pseudo-version → arrastaria core p/ pré-release, viola pin); v1 v0.69.0 existe mas deprecated + driver v1. Padrão: mirror dos outros helpers, `SetMonitor` no `*options.ClientOptions` v2. Verificado no proxy Go 2026-07-22. + +## Riscos registrados +- **otelmongo v2** exige mongo-driver v2 na APP — não na lib (a lib só provê o helper). Migração do driver é Fase 3. +- **gRPC naming:** mantido experimental `rpc.server.duration`+`rpc.grpc.status_code` p/ consistir c/ span; revisar em lockstep quando semconv migrar. diff --git a/docs/metrics-contract.md b/docs/metrics-contract.md new file mode 100644 index 0000000..7fb5612 --- /dev/null +++ b/docs/metrics-contract.md @@ -0,0 +1,57 @@ +# Metric Contract — lib-observability + +> Fonte canônica dos nomes/unidade/buckets/labels das métricas emitidas pela lib. +> Toda métrica nativa DEVE seguir este contrato. Testes validam contra ele. +> Base: OpenTelemetry Semantic Conventions. Ver pre-dev `docs/pre-dev/observability-metrics-standardization/`. + +## Princípios + +1. **Unidade = segundos (`s`)** para toda duração (semconv). NUNCA milissegundos na lib. + - Leitura em ms é responsabilidade do PAINEL: no Grafana, unidade do campo = `seconds (s)` → formata "50 ms"/"1.2 s" automaticamente. Fallback em query: `... * 1000` (fator 1000, multiplica; NÃO converte o label `le` de heatmap/bucket). + - NÃO dual-emitir (ms+s) na lib. NÃO converter no collector. Compat legacy de dashboards antigos migra p/ unidade-do-painel. +2. **Instrumento criado UMA vez** na construção (nunca por request). Record é **no-op** quando o instrumento é nil (telemetria desabilitada) — chamável incondicionalmente, nunca panic, nunca afeta o request path. +3. **Só labels de baixa cardinalidade bounded.** Ver lista PROIBIDA abaixo. Cada valor distinto de um label multiplica séries. +4. **Nomes = OTel semconv estável.** Não inventar chaves; reusar `constants/opentelemetry.go`. +5. **tenant.id** auto em RED (HTTP/gRPC) via resolver existente; MANUAL em métrica de negócio. + +## Buckets advisory (por sinal) + +| Sinal | Buckets (segundos) | +|---|---| +| HTTP / RPC / Messaging | `0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10` | +| Database | `0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10` (mais fino no low-end) | + +## Catálogo (nome / tipo / unidade / labels) + +| Métrica | Tipo | Unidade | Labels permitidos | Estabilidade | +|---|---|---|---|---| +| `http.server.request.duration` | Histogram | s | http.request.method, http.response.status_code, http.route, error.type, tenant.id | STABLE (já existe) | +| `http.client.request.duration` | Histogram | s | http.request.method, http.response.status_code, server.address, error.type | STABLE (já existe) | +| `http.server.active_requests` | UpDownCounter | {request} | http.request.method, http.route (opcional) | (T4) | +| `rpc.server.duration`¹ | Histogram | s | rpc.system, rpc.method, rpc.grpc.status_code, error.type, tenant.id | (T2) | +| `rpc.client.duration`¹ | Histogram | s | rpc.system, rpc.method, rpc.grpc.status_code, error.type | (T2) | +| `db.client.operation.duration` | Histogram | s | db.system.name, db.operation.name, db.collection.name, db.namespace, error.type | STABLE (Fase 2) | +| `messaging.client.operation.duration` (produce) | Histogram | s | messaging.system, messaging.operation.name, messaging.destination.template, error.type | (Fase 2) | +| `messaging.process.duration` (consume) | Histogram | s | messaging.system, messaging.operation.name, messaging.destination.template, messaging.consumer.group.name, error.type | (Fase 2) | +| `go.*` (runtime) | Gauge/Counter/Hist | várias | (dimensões fixas do contrib/runtime) | (T3) | + +¹ **Nota gRPC (validar na T2):** o train contrib atual pode emitir `rpc.server.duration` (experimental, este contrato) ou `rpc.server.call.duration` (RC, semconv). Confirmar o nome contra o pacote pinado no momento da T2 e alinhar. Manter `rpc.grpc.status_code` OU `rpc.response.status_code` conforme o que o interceptor da lib emite (hoje o span/métrica usam `rpc.grpc.status_code`, setado em `grpcmiddleware.WithTelemetryInterceptor` / `recordRPCDuration`). + +## PROIBIDO como label (PII / cardinalidade ilimitada) + +- query text / SQL / bind params / valores de coluna +- `db.query.text` → só em SPAN, opt-in; NUNCA em métrica +- routing key / message id / partition com id +- `url.path` com id/uuid; path resolvido concreto (span.name — raiz do problema original) +- pix key, document (cpf/cnpj), email, qualquer PII +- request/response payload + +## Habilitação / no-op + +- Métrica emitida quando telemetria habilitada (provider+MetricsFactory não-nil — sinal existente na lib) e o subsistema ligado. +- Toggles opt-in em `TelemetryConfig` p/ subsistemas novos (runtime, db-instrumentation), default-safe, degradam p/ no-op quando off. Nunca erro que quebre a app. + +## Referências +- Padrão de implementação (template): `middleware/telemetry.go` — `newHTTPServerDurationHistogram` (:45-61), `recordHTTPServerDuration` (:386+). +- semconv: opentelemetry.io/docs/specs/semconv/{http,rpc,database,messaging}/* +- Decisões: `docs/pre-dev/observability-metrics-standardization/trd.md` (ADRs), `dependency-map.md` (versões). diff --git a/go.mod b/go.mod index d86a166..fe82ff5 100644 --- a/go.mod +++ b/go.mod @@ -3,12 +3,16 @@ module github.com/LerianStudio/lib-observability/v2 go 1.26.3 require ( + github.com/XSAM/otelsql v0.43.0 github.com/gofiber/fiber/v3 v3.4.0 github.com/google/uuid v1.6.0 + github.com/redis/go-redis/extra/redisotel/v9 v9.17.2 + github.com/redis/go-redis/v9 v9.17.2 github.com/shirou/gopsutil v3.21.11+incompatible github.com/shopspring/decimal v1.4.0 github.com/stretchr/testify v1.11.1 go.opentelemetry.io/contrib/bridges/otelzap v0.19.0 + go.opentelemetry.io/contrib/instrumentation/runtime v0.69.0 go.opentelemetry.io/otel v1.44.0 go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0 @@ -22,7 +26,7 @@ require ( go.opentelemetry.io/otel/trace v1.44.0 go.uber.org/mock v0.6.0 go.uber.org/zap v1.28.0 - google.golang.org/grpc v1.81.1 + google.golang.org/grpc v1.82.1 ) require ( @@ -30,6 +34,7 @@ require ( github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect + github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect github.com/go-logr/logr v1.4.3 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-ole/go-ole v1.3.0 // indirect @@ -41,6 +46,7 @@ require ( github.com/mattn/go-isatty v0.0.22 // indirect github.com/philhofer/fwd v1.2.0 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect + github.com/redis/go-redis/extra/rediscmd/v9 v9.17.2 // indirect github.com/tinylib/msgp v1.6.4 // indirect github.com/tklauser/go-sysconf v0.4.0 // indirect github.com/tklauser/numcpus v0.12.0 // indirect @@ -52,7 +58,7 @@ require ( go.uber.org/multierr v1.11.0 // indirect golang.org/x/crypto v0.53.0 // indirect golang.org/x/net v0.56.0 // indirect - golang.org/x/sys v0.46.0 // indirect + golang.org/x/sys v0.47.0 // indirect golang.org/x/text v0.39.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect diff --git a/go.sum b/go.sum index 7de0f88..20d9362 100644 --- a/go.sum +++ b/go.sum @@ -1,11 +1,19 @@ +github.com/XSAM/otelsql v0.43.0 h1:ZIhXqRoMhILXQwBQoq/Dl6Taap/KEFQXZrWjYV1L8X8= +github.com/XSAM/otelsql v0.43.0/go.mod h1:DJBGBvbtwf1OCBYRTjpRFxOqi6ONpdfb+htr4ncRWuw= github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM= github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY= +github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs= +github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c= +github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA= +github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0= github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM= github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78= +github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc= github.com/fxamacker/cbor/v2 v2.9.2 h1:X4Ksno9+x3cz0TZv69ec1hxP/+tymuR8PXQJyDwfh78= github.com/fxamacker/cbor/v2 v2.9.2/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= @@ -44,6 +52,12 @@ github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM= github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/redis/go-redis/extra/rediscmd/v9 v9.17.2 h1:KYWnHK9pwzOUo3sNJlNmzRwZ5mw7opugn8njtGThKNg= +github.com/redis/go-redis/extra/rediscmd/v9 v9.17.2/go.mod h1:wsfMQVl/GFYD9Gx/tlxurlTtvHkZRAt8j1qi27eIlTk= +github.com/redis/go-redis/extra/redisotel/v9 v9.17.2 h1:wthFPRW3Y50CknMrjjJoYwXUFR4U7hMVJCMeLzDI8s4= +github.com/redis/go-redis/extra/redisotel/v9 v9.17.2/go.mod h1:iqfQX7U2o8MWSl8W+Ah8KqbQyi/UoR/MQNgvaUyA1wc= +github.com/redis/go-redis/v9 v9.17.2 h1:P2EGsA4qVIM3Pp+aPocCJ7DguDHhqrXNhVcEp4ViluI= +github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/shamaton/msgpack/v3 v3.1.2 h1:d5gWAIyMU4M0WgDjz6IFSCuXJUA2dFwRHBpDclE8CLw= @@ -74,6 +88,8 @@ go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= go.opentelemetry.io/contrib/bridges/otelzap v0.19.0 h1:48Eq3xxFx2KlL/tF7lnl42kKJBDlhNTLRzv0h154JnM= go.opentelemetry.io/contrib/bridges/otelzap v0.19.0/go.mod h1:cQbV77F0u6HmtZPiQD9oxp2esaOEb4uLqIta6OFIKOk= +go.opentelemetry.io/contrib/instrumentation/runtime v0.69.0 h1:MtkMsuRo3zEXTTMALfyrszwCDZTkB6wolyPjbwFAdq0= +go.opentelemetry.io/contrib/instrumentation/runtime v0.69.0/go.mod h1:FYTxnpsm+UPD0erZNq20GvnM8T2YQHiHtT2vokdpoac= go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU= go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc= go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0 h1:rydZ9sxbcFdm/oWrVyfLTjHIygMgv0bEeMd+3B/BvoM= @@ -120,8 +136,8 @@ golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o= golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= -golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus= golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= @@ -130,8 +146,8 @@ google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1: google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY= google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= -google.golang.org/grpc v1.81.1 h1:VnnIIZ88UzOOKLukQi+ImGz8O1Wdp8nAGGnvOfEIWQQ= -google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I= +google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE= +google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/grpcmiddleware/harness_test.go b/grpcmiddleware/harness_test.go new file mode 100644 index 0000000..c462e2b --- /dev/null +++ b/grpcmiddleware/harness_test.go @@ -0,0 +1,89 @@ +//go:build unit + +package grpcmiddleware + +import ( + "context" + "testing" + + "github.com/LerianStudio/lib-observability/v2/metrics" + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/propagation" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" +) + +// newMetricsHarness wires a real OTel SDK ManualReader so tests can assert on +// the rpc.*.duration histograms exactly as they would appear to an exporter. +// Returns the configured Telemetry pointer plus the reader. +func newMetricsHarness(t *testing.T) (*tracing.Telemetry, *sdkmetric.ManualReader) { + t.Helper() + + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + factory, err := metrics.NewMetricsFactory(mp.Meter("test-library"), nil) + require.NoError(t, err) + + tel := &tracing.Telemetry{ + TelemetryConfig: tracing.TelemetryConfig{ + LibraryName: "test-library", + EnableTelemetry: true, + }, + MeterProvider: mp, + MetricsFactory: factory, + } + + return tel, reader +} + +// newTelemetryHarness extends newMetricsHarness with a real TracerProvider +// backed by an InMemoryExporter so tests can assert on both the duration +// histograms and the span attributes produced by the interceptors. +func newTelemetryHarness( + t *testing.T, +) (*tracing.Telemetry, *sdkmetric.ManualReader, *tracetest.InMemoryExporter) { + t.Helper() + + tel, reader := newMetricsHarness(t) + + spanExp := tracetest.NewInMemoryExporter() + tp := sdktrace.NewTracerProvider(sdktrace.WithSyncer(spanExp)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + + tel.TracerProvider = tp + + return tel, reader, spanExp +} + +// setupTestTracer sets up a test tracer provider and returns it along with a span recorder. +func setupTestTracer(t *testing.T) (*sdktrace.TracerProvider, *tracetest.SpanRecorder) { + t.Helper() + + spanRecorder := tracetest.NewSpanRecorder() + tracerProvider := sdktrace.NewTracerProvider( + sdktrace.WithSpanProcessor(spanRecorder), + ) + + oldPropagator := otel.GetTextMapPropagator() + otel.SetTextMapPropagator(propagation.TraceContext{}) + t.Cleanup(func() { + otel.SetTextMapPropagator(oldPropagator) + }) + + return tracerProvider, spanRecorder +} + +func attrValue(set attribute.Set, key string) (string, bool) { + v, ok := set.Value(attribute.Key(key)) + if !ok { + return "", false + } + + return v.AsString(), true +} diff --git a/grpcmiddleware/telemetry.go b/grpcmiddleware/telemetry.go new file mode 100644 index 0000000..0532ac6 --- /dev/null +++ b/grpcmiddleware/telemetry.go @@ -0,0 +1,620 @@ +// Package grpcmiddleware provides gRPC telemetry interceptors (server and +// client) that integrate with the lib-observability tracing and metrics +// packages. +// +// It is deliberately Fiber-free: none of the code in this package imports +// github.com/gofiber/fiber, so applications still on Fiber v2 can wire up gRPC +// tracing and the rpc.server.duration / rpc.client.duration metrics without +// pulling in Fiber v3. The HTTP counterpart lives in the middleware package, +// and both share the single process-wide system-metrics collector via +// telemetrycore. +package grpcmiddleware + +import ( + "context" + "errors" + "reflect" + "regexp" + "strings" + "sync" + "time" + + observability "github.com/LerianStudio/lib-observability/v2" + constant "github.com/LerianStudio/lib-observability/v2/constants" + "github.com/LerianStudio/lib-observability/v2/telemetrycore" + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/google/uuid" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + "go.opentelemetry.io/otel/trace" + "google.golang.org/grpc" + grpccodes "google.golang.org/grpc/codes" + "google.golang.org/grpc/metadata" + "google.golang.org/grpc/status" +) + +// rpcServerDurationMetric / rpcClientDurationMetric are the metric names for +// gRPC server- and client-side call duration. Recorded as Float64 histograms in +// seconds. +// +// Naming note (see docs/metrics-contract.md): the OTel RPC semconv train is +// mid-migration between the experimental `rpc.server.duration` and the RC +// `rpc.server.call.duration`. We intentionally keep the experimental names so +// the metric aligns with the span attributes this library already emits +// (`rpc.grpc.status_code`), giving operators one consistent RPC vocabulary +// across traces and metrics. Revisit in lockstep when the span attributes +// migrate to the RC names. +const ( + rpcServerDurationMetric = "rpc.server.duration" + rpcClientDurationMetric = "rpc.client.duration" +) + +// rpcSystemGRPC is the rpc.system attribute value for gRPC calls. +const rpcSystemGRPC = "grpc" + +// metadataID is the gRPC metadata key that carries the request context identifier. +const metadataID = "metadata_id" + +// headerUserAgent is the HTTP/gRPC User-Agent header key. +const headerUserAgent = "User-Agent" + +// ErrContextNotFound is returned when a required context is nil. +var ErrContextNotFound = errors.New("context not found") + +// internalServicePattern matches Lerian internal service user-agent strings. +var internalServicePattern = regexp.MustCompile(`^[\w-]+/[\d.]+\s+LerianStudio$`) + +// rpcDurationBuckets follows the current OpenTelemetry advisory layout shared by +// the HTTP and RPC signals (docs/metrics-contract.md). Update only in lockstep +// with the spec. +var rpcDurationBuckets = []float64{ + 0.005, 0.01, 0.025, 0.05, 0.075, + 0.1, 0.25, 0.5, 0.75, + 1, 2.5, 5, 7.5, 10, +} + +// newRPCDurationHistogram builds a float64 seconds histogram for the given RPC +// duration metric name on the provided meter. Returns nil if the meter is nil +// or instrument creation fails - callers must treat nil as "do not record". +func newRPCDurationHistogram(meter metric.Meter, name, description string) metric.Float64Histogram { + if meter == nil { + return nil + } + + hist, err := meter.Float64Histogram( + name, + metric.WithUnit("s"), + metric.WithDescription(description), + metric.WithExplicitBucketBoundaries(rpcDurationBuckets...), + ) + if err != nil { + return nil + } + + return hist +} + +// classifyGRPCErrorType returns the low-cardinality error.type label for the +// RPC duration metrics. Any non-OK gRPC status maps to the canonical code name +// (a bounded enum, e.g. "NotFound", "Unavailable"), and OK maps to "" so +// successful calls carry no error.type. Using the code name rather than the +// handler's Go error type keeps the label set bounded regardless of how many +// distinct application errors flow through. +func classifyGRPCErrorType(code grpccodes.Code) string { + if code == grpccodes.OK { + return "" + } + + return code.String() +} + +type spanEndStateKey struct{} + +type spanEndState struct { + span trace.Span + // once guarantees End() is idempotent. It still protects the foreign/handler + // -created span on the gRPC fallback path (where owned==false and the span + // may be ended both by a defer and by the End interceptor). owned resolves + // ordering (which interceptor ends the span, and that it happens after + // finalization); once resolves double-end. + once sync.Once + // owned marks the span as exclusively finalized/ended by the interceptor + // that created it (WithTelemetryInterceptor). When set, + // EndTracingSpansInterceptor must NOT end it: the owning interceptor ends it + // via its own deferred End() AFTER applying rpc.method / status attributes. + owned bool +} + +func newSpanEndState(span trace.Span) *spanEndState { + return &spanEndState{span: span} +} + +func (s *spanEndState) End() { + if s == nil || s.span == nil { + return + } + + s.once.Do(func() { s.span.End() }) +} + +func contextWithSpanEndState(ctx context.Context, state *spanEndState) context.Context { + if ctx == nil { + ctx = context.Background() + } + + return context.WithValue(ctx, spanEndStateKey{}, state) +} + +func spanEndStateFromContext(ctx context.Context) *spanEndState { + if ctx == nil { + return nil + } + + state, _ := ctx.Value(spanEndStateKey{}).(*spanEndState) + + return state +} + +// TelemetryMiddleware wraps gRPC handlers with tracing and metrics setup. +type TelemetryMiddleware struct { + Telemetry *tracing.Telemetry +} + +// NewTelemetryMiddleware creates a new instance of TelemetryMiddleware. +func NewTelemetryMiddleware(tl *tracing.Telemetry) *TelemetryMiddleware { + return &TelemetryMiddleware{tl} +} + +// collectMetrics ensures the background metrics collector goroutine is running. +// It delegates to telemetrycore so the gRPC interceptors and the HTTP +// middleware share a single collector singleton. +func (tm *TelemetryMiddleware) collectMetrics(_ context.Context) error { + if tm == nil { + return nil + } + + return telemetrycore.EnsureMetricsCollector(tm.Telemetry) +} + +// WithTelemetryInterceptor is a gRPC interceptor that adds tracing to the context. +// +// When the effective Telemetry has a non-nil MeterProvider AND a non-nil +// MetricsFactory, the interceptor also records the rpc.server.duration +// (Float64 seconds) histogram for every call, independently of whether tracing +// is enabled - mirroring the HTTP WithTelemetry gate. Recording is best-effort: +// nil telemetry / MeterProvider / MetricsFactory and instrument-creation errors +// all silently skip the metric without affecting the request path. +func (tm *TelemetryMiddleware) WithTelemetryInterceptor(tl *tracing.Telemetry) grpc.UnaryServerInterceptor { + // Build the server duration histogram once at interceptor-construction time, + // symmetric to the HTTP WithTelemetry construction-once block. + var serverDurationHistogram metric.Float64Histogram + + bootstrapTelemetry := tl + if bootstrapTelemetry == nil && tm != nil { + bootstrapTelemetry = tm.Telemetry + } + + if bootstrapTelemetry != nil && + bootstrapTelemetry.MeterProvider != nil && + bootstrapTelemetry.MetricsFactory != nil { + serverDurationHistogram = newRPCDurationHistogram( + bootstrapTelemetry.MeterProvider.Meter(bootstrapTelemetry.LibraryName), + rpcServerDurationMetric, + "Duration of gRPC server calls.", + ) + } + + return func( + ctx context.Context, + req any, + info *grpc.UnaryServerInfo, + handler grpc.UnaryHandler, + ) (any, error) { + ctx = normalizeGRPCContext(ctx) + + effectiveTelemetry := tl + if effectiveTelemetry == nil && tm != nil { + effectiveTelemetry = tm.Telemetry + } + + if effectiveTelemetry == nil { + return handler(ctx, req) + } + + requestID := resolveGRPCRequestID(ctx, req) + ctx = observability.ContextWithHeaderID(ctx, requestID) + + methodName := "unknown" + if info != nil { + methodName = info.FullMethod + } + + if effectiveTelemetry.TracerProvider == nil { + start := time.Now() + resp, err := handler(ctx, req) + + recordRPCDuration(ctx, serverDurationHistogram, methodName, start, err, + ResolveTenantIDFromGRPC(ctx)) + + return resp, err + } + + tracer := effectiveTelemetry.TracerProvider.Tracer(effectiveTelemetry.LibraryName) + + tenantID := ResolveTenantIDFromGRPC(ctx) + if tenantID != "" { + ctx = observability.ContextWithSpanAttributes(ctx, attribute.String(constant.AttrKeyTenantID, tenantID)) + } + + ctx = observability.ContextWithSpanAttributes(ctx, + attribute.String("app.request.request_id", requestID), + attribute.String("grpc.method", methodName), + ) + + traceCtx := ctx + // Compatibility note: trace extraction currently trusts the internal-service + // User-Agent heuristic. This is an interoperability hint, not an authenticated + // trust boundary, and is preserved to avoid changing existing caller behavior. + if isInternalLerianService(getGRPCUserAgent(ctx)) { + md, _ := metadata.FromIncomingContext(ctx) + traceCtx = tracing.ExtractGRPCContext(ctx, md) + } + + ctx, span := tracer.Start(traceCtx, methodName, trace.WithSpanKind(trace.SpanKindServer)) + endState := newSpanEndState(span) + // WithTelemetryInterceptor owns this span's lifecycle: it applies + // rpc.method / rpc.grpc.status_code / handler error status AFTER the + // handler returns (below), then ends the span via the defer. Marking it + // owned makes EndTracingSpansInterceptor skip it, so those post-handler + // attributes can't be dropped by a chain where the end interceptor + // unwinds first — mirroring the HTTP WithTelemetry/EndTracingSpans pair. + endState.owned = true + + defer endState.End() + + ctx = observability.ContextWithTracer(ctx, tracer) + ctx = observability.ContextWithMetricFactory(ctx, effectiveTelemetry.MetricsFactory) + ctx = contextWithSpanEndState(ctx, endState) + + err := tm.collectMetrics(ctx) + if err != nil { + tracing.HandleSpanError(span, "Failed to collect metrics", err) + } + + // Capture start immediately before the handler so the duration metric + // reflects the handler chain, then record after the status is known. + start := time.Now() + resp, err := handler(ctx, req) + + grpcStatusCode := status.Code(err) + span.SetAttributes( + attribute.String("rpc.method", methodName), + attribute.Int("rpc.grpc.status_code", int(grpcStatusCode)), + ) + + if err != nil { + tracing.HandleSpanError(span, "gRPC handler error", err) + } + + recordRPCDuration(ctx, serverDurationHistogram, methodName, start, err, tenantID) + + return resp, err + } +} + +// recordRPCDuration emits an RPC duration histogram observation (server or +// client) for a completed unary call. It is a no-op when the histogram is nil +// (telemetry / MeterProvider / MetricsFactory absent or instrument creation +// failed), so callers can invoke it unconditionally. +// +// Attribute set follows the metric contract (docs/metrics-contract.md): +// - rpc.system: always "grpc" +// - rpc.method: the gRPC full method (bounded set of registered methods) +// - rpc.grpc.status_code: the numeric gRPC status code, matching the span +// attribute this library already emits +// - error.type: only set for non-OK statuses, using the canonical code name +// (a bounded enum) to keep cardinality low +// - tenant.id: server-side only, passed by the caller (already resolved via +// ResolveTenantIDFromGRPC); the client path passes "" so the label is +// omitted, since a client does not own the tenant boundary +func recordRPCDuration( + ctx context.Context, + hist metric.Float64Histogram, + methodName string, + start time.Time, + callErr error, + tenantID string, +) { + if hist == nil { + return + } + + grpcStatusCode := status.Code(callErr) + + attrs := []attribute.KeyValue{ + attribute.String("rpc.system", rpcSystemGRPC), + attribute.String("rpc.method", methodName), + attribute.Int("rpc.grpc.status_code", int(grpcStatusCode)), + } + + if errType := classifyGRPCErrorType(grpcStatusCode); errType != "" { + attrs = append(attrs, attribute.String("error.type", errType)) + } + + if tenantID != "" { + attrs = append(attrs, attribute.String(constant.AttrKeyTenantID, tenantID)) + } + + hist.Record(ctx, time.Since(start).Seconds(), metric.WithAttributes(attrs...)) +} + +// EndTracingSpansInterceptor is a gRPC interceptor that ends the tracing spans. +func (tm *TelemetryMiddleware) EndTracingSpansInterceptor() grpc.UnaryServerInterceptor { + return func( + ctx context.Context, + req any, + info *grpc.UnaryServerInfo, + handler grpc.UnaryHandler, + ) (any, error) { + resp, err := handler(ctx, req) + if state := spanEndStateFromContext(ctx); state != nil { + // A span owned by WithTelemetryInterceptor is finalized and ended by + // that interceptor after it records post-handler attributes/status. + // Skip it here (same reasoning as the HTTP EndTracingSpans). + if state.owned { + return resp, err + } + + state.End() + + return resp, err + } + + trace.SpanFromContext(ctx).End() + + return resp, err + } +} + +// UnaryClientInterceptor is a gRPC client interceptor that propagates trace +// context on outgoing calls and records the rpc.client.duration (Float64 +// seconds) histogram. +// +// The histogram is built once at construction time and gated on a non-nil +// MeterProvider AND MetricsFactory, mirroring the server interceptor. Trace +// context is injected into the outgoing metadata via tracing.InjectGRPCContext +// so downstream services join the trace instead of starting a new root. +// Recording and injection are best-effort: nil telemetry degrades to a plain +// pass-through to the invoker, never blocking the call. +// +// The client metric intentionally OMITS tenant.id: a client does not own the +// tenant boundary, and the server side already attributes the call to a tenant. +func (tm *TelemetryMiddleware) UnaryClientInterceptor(tl *tracing.Telemetry) grpc.UnaryClientInterceptor { + var clientDurationHistogram metric.Float64Histogram + + bootstrapTelemetry := tl + if bootstrapTelemetry == nil && tm != nil { + bootstrapTelemetry = tm.Telemetry + } + + if bootstrapTelemetry != nil && + bootstrapTelemetry.MeterProvider != nil && + bootstrapTelemetry.MetricsFactory != nil { + clientDurationHistogram = newRPCDurationHistogram( + bootstrapTelemetry.MeterProvider.Meter(bootstrapTelemetry.LibraryName), + rpcClientDurationMetric, + "Duration of gRPC client calls.", + ) + } + + return func( + ctx context.Context, + method string, + req, reply any, + cc *grpc.ClientConn, + invoker grpc.UnaryInvoker, + opts ...grpc.CallOption, + ) error { + ctx = normalizeGRPCContext(ctx) + + // Inject trace context into the outgoing metadata so the downstream + // server joins this trace. Merge with any metadata already on the + // outgoing context rather than overwriting it. + md, ok := metadata.FromOutgoingContext(ctx) + if !ok || md == nil { + md = metadata.New(nil) + } else { + md = md.Copy() + } + + md = tracing.InjectGRPCContext(ctx, md) + ctx = metadata.NewOutgoingContext(ctx, md) + + start := time.Now() + err := invoker(ctx, method, req, reply, cc, opts...) + + // Client metric carries no tenant.id (empty string omits the label). + recordRPCDuration(ctx, clientDurationHistogram, method, start, err, "") + + return err + } +} + +// resolveGRPCRequestID determines the request ID for a gRPC call from the request body, +// existing context header, gRPC metadata (in that priority order), or generates a new UUID. +func resolveGRPCRequestID(ctx context.Context, req any) string { + if rid, ok := getValidBodyRequestID(req); ok { + return rid + } + + if existing := getContextHeaderID(ctx); existing != "" { + return existing + } + + if rid := getMetadataID(ctx); rid != "" { + return rid + } + + return uuid.New().String() +} + +// getContextHeaderID extracts the HeaderID from the observability context value. +func getContextHeaderID(ctx context.Context) string { + if ctx == nil { + return "" + } + + cv, ok := ctx.Value(observability.ContextKey).(*observability.ContextValue) + if !ok || cv == nil { + return "" + } + + return normalizeRequestID(cv.HeaderID) +} + +// getValidBodyRequestID extracts and validates the request_id from the gRPC request body. +// Returns (id, true) when present and valid UUID; otherwise ("", false). +func getValidBodyRequestID(req any) (string, bool) { + if req == nil { + return "", false + } + + // Check for typed-nil interface. + v := reflect.ValueOf(req) + if (v.Kind() == reflect.Pointer || v.Kind() == reflect.Interface) && v.IsNil() { + return "", false + } + + r, ok := req.(interface{ GetRequestId() string }) + if !ok { + return "", false + } + + rid := strings.TrimSpace(r.GetRequestId()) + if rid == "" { + return "", false + } + + // Validate it is a UUID. + if _, err := uuid.Parse(rid); err != nil { + return "", false + } + + return rid, true +} + +// normalizeRequestID trims whitespace and control characters from a raw request ID. +func normalizeRequestID(raw string) string { + return strings.TrimSpace(sanitizeLogValue(raw)) +} + +// sanitizeLogValue strips ASCII control bytes (log-injection defense) from raw. +func sanitizeLogValue(raw string) string { + return strings.Map(func(r rune) rune { + if r < 0x20 || r == 0x7f { + return -1 + } + + return r + }, raw) +} + +// isNilOrEmptyString reports whether a string pointer is nil or the trimmed +// value is empty. "null" and "nil" are treated as empty to handle JSON null +// serialization artifacts where some encoders emit the literal string "null" +// or "nil" instead of a JSON null. +func isNilOrEmptyString(s *string) bool { + return s == nil || strings.TrimSpace(*s) == "" || strings.TrimSpace(*s) == "null" || strings.TrimSpace(*s) == "nil" +} + +// normalizeGRPCContext returns a non-nil context, falling back to context.Background(). +func normalizeGRPCContext(ctx context.Context) context.Context { + if ctx == nil { + return context.Background() + } + + return ctx +} + +// getMetadataID extracts a correlation id from incoming gRPC metadata if present. +func getMetadataID(ctx context.Context) string { + if ctx == nil { + return "" + } + + if md, ok := metadata.FromIncomingContext(ctx); ok && md != nil { + headerID := md.Get(metadataID) + if len(headerID) > 0 { + v := normalizeRequestID(headerID[0]) + if v != "" && v != "null" && v != "nil" { + return v + } + } + } + + return "" +} + +// getGRPCUserAgent extracts the User-Agent from incoming gRPC metadata. +// Returns empty string if the metadata is not present or doesn't contain user-agent. +func getGRPCUserAgent(ctx context.Context) string { + if ctx == nil { + return "" + } + + md, ok := metadata.FromIncomingContext(ctx) + if !ok || md == nil { + return "" + } + + userAgents := md.Get(strings.ToLower(headerUserAgent)) + if len(userAgents) == 0 { + return "" + } + + return userAgents[0] +} + +// isInternalLerianService reports whether a user-agent belongs to a Lerian internal service. +func isInternalLerianService(userAgent string) bool { + return internalServicePattern.MatchString(userAgent) +} + +// ResolveTenantIDFromGRPC returns the tenant identifier carried by the +// canonical tenant-id gRPC metadata key, normalized for safe inclusion in +// telemetry. Returns an empty string when the metadata is absent, empty, or +// longer than MaxTenantIDLen bytes. The metadata is trusted only as an +// observability hint: callers MUST authenticate the tenant separately. +func ResolveTenantIDFromGRPC(ctx context.Context) string { + if ctx == nil { + return "" + } + + md, ok := metadata.FromIncomingContext(ctx) + if !ok || md == nil { + return "" + } + + vals := md.Get(constant.MetadataTenantID) + if len(vals) == 0 { + return "" + } + + return sanitizeTenantID(vals[0]) +} + +// sanitizeTenantID trims whitespace and control bytes from raw, then enforces +// the MaxTenantIDLen cap. Returns "" for any value that fails normalization or +// exceeds the cap, so callers can use it as a presence check. +func sanitizeTenantID(raw string) string { + value := normalizeRequestID(raw) + if isNilOrEmptyString(&value) { + return "" + } + + if len(value) > constant.MaxTenantIDLen { + return "" + } + + return value +} diff --git a/grpcmiddleware/telemetry_grpc_metrics_test.go b/grpcmiddleware/telemetry_grpc_metrics_test.go new file mode 100644 index 0000000..961172d --- /dev/null +++ b/grpcmiddleware/telemetry_grpc_metrics_test.go @@ -0,0 +1,298 @@ +//go:build unit + +package grpcmiddleware + +import ( + "context" + "errors" + "testing" + + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "google.golang.org/grpc" + grpccodes "google.golang.org/grpc/codes" + "google.golang.org/grpc/metadata" + "google.golang.org/grpc/status" +) + +// findRPCDurationHistogram extracts a named RPC duration histogram data point +// from a ManualReader collection. Returns nil if the metric is absent (used to +// assert non-recording paths). When present it also locks the unit to seconds, +// matching the metric contract (docs/metrics-contract.md). +func findRPCDurationHistogram( + t *testing.T, + reader *sdkmetric.ManualReader, + metricName string, +) *metricdata.HistogramDataPoint[float64] { + t.Helper() + + rm := &metricdata.ResourceMetrics{} + require.NoError(t, reader.Collect(context.Background(), rm)) + + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name != metricName { + continue + } + + h, ok := m.Data.(metricdata.Histogram[float64]) + require.True(t, ok, "expected Float64 histogram for %s, got %T", m.Name, m.Data) + require.NotEmpty(t, h.DataPoints, "histogram has no data points") + require.Equal(t, "s", m.Unit, "metric unit must be seconds") + + dp := h.DataPoints[0] + return &dp + } + } + + return nil +} + +// TestRPCServerDurationBuckets_MatchOTelAdvisory locks the RPC bucket layout +// against the shared HTTP/RPC advisory (metric contract). Any change is +// observable from dashboards, so it MUST be a deliberate spec-tracking update. +func TestRPCServerDurationBuckets_MatchOTelAdvisory(t *testing.T) { + expected := []float64{ + 0.005, 0.01, 0.025, 0.05, 0.075, + 0.1, 0.25, 0.5, 0.75, + 1, 2.5, 5, 7.5, 10, + } + assert.Equal(t, expected, rpcDurationBuckets) +} + +// TestWithTelemetryInterceptor_RecordsServerDurationOnSuccess verifies a +// successful unary call emits rpc.server.duration with rpc.system=grpc, +// rpc.method, and rpc.grpc.status_code=0 (OK), and NO error.type. +func TestWithTelemetryInterceptor_RecordsServerDurationOnSuccess(t *testing.T) { + tel, reader := newMetricsHarness(t) + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.WithTelemetryInterceptor(tel) + + handler := func(_ context.Context, _ any) (any, error) { + return "ok", nil + } + info := &grpc.UnaryServerInfo{FullMethod: "/test.Service/DoThing"} + + _, err := interceptor(context.Background(), "req", info, handler) + require.NoError(t, err) + + dp := findRPCDurationHistogram(t, reader, rpcServerDurationMetric) + require.NotNil(t, dp, "expected rpc.server.duration to be recorded") + assert.EqualValues(t, 1, dp.Count) + assert.GreaterOrEqual(t, dp.Sum, 0.0, "duration sum must be non-negative seconds") + + system, ok := attrValue(dp.Attributes, "rpc.system") + require.True(t, ok) + assert.Equal(t, "grpc", system) + + method, ok := attrValue(dp.Attributes, "rpc.method") + require.True(t, ok) + assert.Equal(t, "/test.Service/DoThing", method) + + code, ok := dp.Attributes.Value(attribute.Key("rpc.grpc.status_code")) + require.True(t, ok) + assert.EqualValues(t, int(grpccodes.OK), code.AsInt64()) + + _, hasErr := dp.Attributes.Value(attribute.Key("error.type")) + assert.False(t, hasErr, "error.type must be absent on OK responses") +} + +// TestWithTelemetryInterceptor_RecordsServerDurationOnError verifies a failing +// unary call records the numeric gRPC status code and a low-cardinality +// error.type derived from that code. +func TestWithTelemetryInterceptor_RecordsServerDurationOnError(t *testing.T) { + tel, reader := newMetricsHarness(t) + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.WithTelemetryInterceptor(tel) + + handler := func(_ context.Context, _ any) (any, error) { + return nil, status.Error(grpccodes.NotFound, "missing") + } + info := &grpc.UnaryServerInfo{FullMethod: "/test.Service/GetThing"} + + _, err := interceptor(context.Background(), "req", info, handler) + require.Error(t, err) + + dp := findRPCDurationHistogram(t, reader, rpcServerDurationMetric) + require.NotNil(t, dp) + + code, ok := dp.Attributes.Value(attribute.Key("rpc.grpc.status_code")) + require.True(t, ok) + assert.EqualValues(t, int(grpccodes.NotFound), code.AsInt64()) + + errType, ok := attrValue(dp.Attributes, "error.type") + require.True(t, ok, "error.type must be set when status != OK") + assert.Equal(t, grpccodes.NotFound.String(), errType) +} + +// TestWithTelemetryInterceptor_RecordsServerDurationWithTenantID verifies the +// tenant.id label is populated from inbound gRPC metadata via the same resolver +// the span uses. +func TestWithTelemetryInterceptor_RecordsServerDurationWithTenantID(t *testing.T) { + tel, reader := newMetricsHarness(t) + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.WithTelemetryInterceptor(tel) + + handler := func(_ context.Context, _ any) (any, error) { return "ok", nil } + info := &grpc.UnaryServerInfo{FullMethod: "/test.Service/DoThing"} + + md := metadata.New(map[string]string{"tenant-id": "acme"}) + ctx := metadata.NewIncomingContext(context.Background(), md) + + _, err := interceptor(ctx, "req", info, handler) + require.NoError(t, err) + + dp := findRPCDurationHistogram(t, reader, rpcServerDurationMetric) + require.NotNil(t, dp) + + tenant, ok := attrValue(dp.Attributes, "tenant.id") + require.True(t, ok, "tenant.id must be present when tenant-id metadata is supplied") + assert.Equal(t, "acme", tenant) +} + +// TestWithTelemetryInterceptor_NilMetricsFactoryDoesNotRecord verifies the +// server metric is gated on MetricsFactory presence, matching the HTTP path. +func TestWithTelemetryInterceptor_NilMetricsFactoryDoesNotRecord(t *testing.T) { + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + tel := &tracing.Telemetry{ + TelemetryConfig: tracing.TelemetryConfig{LibraryName: "test-library"}, + MeterProvider: mp, + // MetricsFactory intentionally nil. + } + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.WithTelemetryInterceptor(tel) + + handler := func(_ context.Context, _ any) (any, error) { return "ok", nil } + info := &grpc.UnaryServerInfo{FullMethod: "/test.Service/DoThing"} + + _, err := interceptor(context.Background(), "req", info, handler) + require.NoError(t, err) + + assert.Nil(t, findRPCDurationHistogram(t, reader, rpcServerDurationMetric), + "nil MetricsFactory must not record rpc.server.duration") +} + +// TestUnaryClientInterceptor_RecordsClientDurationOnSuccess verifies the +// client interceptor emits rpc.client.duration with rpc.system=grpc, +// rpc.method, rpc.grpc.status_code=0 and NO tenant.id (client side omits it). +func TestUnaryClientInterceptor_RecordsClientDurationOnSuccess(t *testing.T) { + tel, reader := newMetricsHarness(t) + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.UnaryClientInterceptor(tel) + + invoker := func(_ context.Context, _ string, _, _ any, _ *grpc.ClientConn, _ ...grpc.CallOption) error { + return nil + } + + err := interceptor(context.Background(), "/test.Service/DoThing", "req", "reply", nil, invoker) + require.NoError(t, err) + + dp := findRPCDurationHistogram(t, reader, rpcClientDurationMetric) + require.NotNil(t, dp, "expected rpc.client.duration to be recorded") + assert.EqualValues(t, 1, dp.Count) + + system, ok := attrValue(dp.Attributes, "rpc.system") + require.True(t, ok) + assert.Equal(t, "grpc", system) + + method, ok := attrValue(dp.Attributes, "rpc.method") + require.True(t, ok) + assert.Equal(t, "/test.Service/DoThing", method) + + code, ok := dp.Attributes.Value(attribute.Key("rpc.grpc.status_code")) + require.True(t, ok) + assert.EqualValues(t, int(grpccodes.OK), code.AsInt64()) + + _, hasErr := dp.Attributes.Value(attribute.Key("error.type")) + assert.False(t, hasErr) + + _, hasTenant := dp.Attributes.Value(attribute.Key("tenant.id")) + assert.False(t, hasTenant, "client metric must never carry tenant.id") +} + +// TestUnaryClientInterceptor_RecordsClientDurationOnError verifies the client +// interceptor records the numeric status code and a numeric-code error.type +// when the invoker returns an error. +func TestUnaryClientInterceptor_RecordsClientDurationOnError(t *testing.T) { + tel, reader := newMetricsHarness(t) + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.UnaryClientInterceptor(tel) + + invoker := func(_ context.Context, _ string, _, _ any, _ *grpc.ClientConn, _ ...grpc.CallOption) error { + return status.Error(grpccodes.Unavailable, "down") + } + + err := interceptor(context.Background(), "/test.Service/DoThing", "req", "reply", nil, invoker) + require.Error(t, err) + + dp := findRPCDurationHistogram(t, reader, rpcClientDurationMetric) + require.NotNil(t, dp) + + code, ok := dp.Attributes.Value(attribute.Key("rpc.grpc.status_code")) + require.True(t, ok) + assert.EqualValues(t, int(grpccodes.Unavailable), code.AsInt64()) + + errType, ok := attrValue(dp.Attributes, "error.type") + require.True(t, ok) + assert.Equal(t, grpccodes.Unavailable.String(), errType) +} + +// TestUnaryClientInterceptor_InjectsTraceContext verifies the client +// interceptor propagates trace context into outgoing gRPC metadata, so +// downstream services join the trace rather than starting a new root. A real +// span is started (and the global W3C propagator installed) so there is a valid +// SpanContext for InjectGRPCContext to serialize. +func TestUnaryClientInterceptor_InjectsTraceContext(t *testing.T) { + tel, _, _ := newTelemetryHarness(t) + + tp, _ := setupTestTracer(t) // installs the global TraceContext propagator + ctx, span := tp.Tracer("client-test").Start(context.Background(), "caller") + defer span.End() + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.UnaryClientInterceptor(tel) + + var outgoingMD metadata.MD + + invoker := func(ictx context.Context, _ string, _, _ any, _ *grpc.ClientConn, _ ...grpc.CallOption) error { + outgoingMD, _ = metadata.FromOutgoingContext(ictx) + return nil + } + + err := interceptor(ctx, "/test.Service/DoThing", "req", "reply", nil, invoker) + require.NoError(t, err) + + require.NotNil(t, outgoingMD, "invoker must observe outgoing metadata") + assert.NotEmpty(t, outgoingMD.Get("traceparent"), + "client interceptor must inject the W3C traceparent for propagation") +} + +// TestUnaryClientInterceptor_NilTelemetryIsNoOp verifies the client interceptor +// is safe with nil telemetry and still invokes the downstream call. +func TestUnaryClientInterceptor_NilTelemetryIsNoOp(t *testing.T) { + mid := NewTelemetryMiddleware(nil) + interceptor := mid.UnaryClientInterceptor(nil) + + called := false + invoker := func(_ context.Context, _ string, _, _ any, _ *grpc.ClientConn, _ ...grpc.CallOption) error { + called = true + return errors.New("downstream") + } + + err := interceptor(context.Background(), "/test.Service/DoThing", "req", "reply", nil, invoker) + require.Error(t, err) + assert.True(t, called, "invoker must be called even when telemetry is nil") +} diff --git a/grpcmiddleware/telemetry_test.go b/grpcmiddleware/telemetry_test.go new file mode 100644 index 0000000..335d9ae --- /dev/null +++ b/grpcmiddleware/telemetry_test.go @@ -0,0 +1,163 @@ +//go:build unit + +package grpcmiddleware + +import ( + "context" + "testing" + + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + oteltrace "go.opentelemetry.io/otel/trace" + "google.golang.org/grpc" + "google.golang.org/grpc/metadata" +) + +// TestWithTelemetryInterceptorConditionalTracePropagation tests conditional trace propagation in gRPC interceptor. +func TestWithTelemetryInterceptorConditionalTracePropagation(t *testing.T) { + tests := []struct { + name string + userAgent string + traceparent string + shouldPropagateTrace bool + description string + }{ + { + name: "Internal Lerian service via gRPC - should propagate trace", + userAgent: "midaz/1.0.0 LerianStudio", + traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", + shouldPropagateTrace: true, + description: "Internal gRPC service should propagate trace context", + }, + { + name: "External gRPC client - should NOT propagate trace", + userAgent: "grpc-go/1.50.0", + traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", + shouldPropagateTrace: false, + description: "External gRPC client should create new root span", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + + tp, spanRecorder := setupTestTracer(t) + defer func() { + _ = tp.Shutdown(ctx) + }() + + oldTracerProvider := otel.GetTracerProvider() + otel.SetTracerProvider(tp) + defer otel.SetTracerProvider(oldTracerProvider) + + tel := &tracing.Telemetry{ + TelemetryConfig: tracing.TelemetryConfig{ + LibraryName: "test-library", + EnableTelemetry: true, + }, + TracerProvider: tp, + } + + mid := NewTelemetryMiddleware(tel) + interceptor := mid.WithTelemetryInterceptor(tel) + + md := metadata.New(map[string]string{}) + if tt.userAgent != "" { + md.Set("user-agent", tt.userAgent) + } + if tt.traceparent != "" { + md.Set("traceparent", tt.traceparent) + } + ctx = metadata.NewIncomingContext(ctx, md) + + var capturedSpanContext oteltrace.SpanContext + handler := func(ctx context.Context, req any) (any, error) { + capturedSpanContext = oteltrace.SpanContextFromContext(ctx) + return "response", nil + } + + info := &grpc.UnaryServerInfo{ + FullMethod: "/test.Service/Method", + } + + _, err := interceptor(ctx, "request", info, handler) + require.NoError(t, err) + + spans := spanRecorder.Ended() + require.GreaterOrEqual(t, len(spans), 1, "Expected at least one span to be created") + + if tt.shouldPropagateTrace { + assert.True(t, capturedSpanContext.IsValid(), "Span context should be valid for internal services") + assert.Equal(t, "4bf92f3577b34da6a3ce929d0e0e4736", capturedSpanContext.TraceID().String(), + "Trace ID should match the traceparent for internal gRPC services") + } else { + require.True(t, capturedSpanContext.IsValid(), "Expected middleware to attach a valid span context") + assert.NotEqual(t, "4bf92f3577b34da6a3ce929d0e0e4736", capturedSpanContext.TraceID().String(), + "Trace ID should be different from traceparent for external services") + } + }) + } +} + +// TestGetGRPCUserAgent tests the getGRPCUserAgent helper function. +func TestGetGRPCUserAgent(t *testing.T) { + tests := []struct { + name string + setupMetadata func() context.Context + expectedUA string + description string + }{ + { + name: "Valid user-agent in metadata", + setupMetadata: func() context.Context { + md := metadata.Pairs("user-agent", "midaz/1.0.0 LerianStudio") + return metadata.NewIncomingContext(context.Background(), md) + }, + expectedUA: "midaz/1.0.0 LerianStudio", + description: "Should extract user-agent from gRPC metadata", + }, + { + name: "No metadata in context", + setupMetadata: func() context.Context { + return context.Background() + }, + expectedUA: "", + description: "Should return empty string when no metadata present", + }, + { + name: "Metadata without user-agent", + setupMetadata: func() context.Context { + md := metadata.Pairs("authorization", "Bearer token") + return metadata.NewIncomingContext(context.Background(), md) + }, + expectedUA: "", + description: "Should return empty string when user-agent key not present", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := tt.setupMetadata() + result := getGRPCUserAgent(ctx) + assert.Equal(t, tt.expectedUA, result, tt.description) + }) + } +} + +// TestEndTracingSpansInterceptor_EndsUnownedSpan verifies the end interceptor +// ends a span that is present in the context but not owned by +// WithTelemetryInterceptor. +func TestEndTracingSpansInterceptor_EndsUnownedSpan(t *testing.T) { + mid := NewTelemetryMiddleware(nil) + end := mid.EndTracingSpansInterceptor() + + handler := func(_ context.Context, _ any) (any, error) { return "ok", nil } + info := &grpc.UnaryServerInfo{FullMethod: "/test.Service/DoThing"} + + resp, err := end(context.Background(), "req", info, handler) + require.NoError(t, err) + assert.Equal(t, "ok", resp) +} diff --git a/messagingobs/doc.go b/messagingobs/doc.go new file mode 100644 index 0000000..16dec3b --- /dev/null +++ b/messagingobs/doc.go @@ -0,0 +1,40 @@ +// Package messagingobs provides thin, nil-safe helpers that instrument RabbitMQ +// producers and consumers with OpenTelemetry spans, trace-context propagation, +// and semantic-convention duration metrics. There is no official RabbitMQ OTel +// instrumentation package, so this is hand-rolled (ADR-006) on top of the +// trace-propagation helpers already in the tracing package +// (InjectTraceHeadersIntoQueue / ExtractTraceContextFromQueueHeaders). +// +// # Boundary (ADR-006, ADR-007) +// +// This package takes NO dependency on an AMQP client. It operates on generic +// header maps (map[string]any, the shape amqp091-go uses for +// amqp.Table/Publishing.Headers and Delivery.Headers). The application owns the +// amqp.Channel: the producer helper returns the headers to attach to the +// outgoing Publishing, and the consumer helper reads the headers off the inbound +// Delivery. This keeps the amqp dependency in the app. +// +// # Emitted telemetry (docs/metrics-contract.md) +// +// - Producer: a producer-kind span, trace context injected into the returned +// headers, and messaging.client.operation.duration (Float64 seconds). +// - Consumer: a consumer-kind span joined to the producer's trace via the +// inbound headers, and messaging.process.duration (Float64 seconds). +// +// Labels are bounded: messaging.system=rabbitmq, messaging.operation.name, +// messaging.destination.template (a TEMPLATE such as "transactions.{tenant}", +// never a concrete queue name or routing key), messaging.consumer.group.name +// (consumer only), and error.type on failures. The concrete routing key and +// message id are accepted only to be attached to the span body via +// RecordError/logs by the caller if desired — they are NEVER emitted as span or +// metric attributes here, per the FORBIDDEN list. +// +// The messaging.* names/units are shared with lib-streaming (RedPanda/Kafka) so +// a single worker dashboard is transport-agnostic. +// +// # No-op degradation (ADR-008) +// +// With nil telemetry (or a nil MeterProvider/MetricsFactory) the helpers still +// return a valid context, injectable headers, and a finish func that is a +// no-op. They never panic and never break the messaging path. +package messagingobs diff --git a/messagingobs/messagingobs.go b/messagingobs/messagingobs.go new file mode 100644 index 0000000..426f71d --- /dev/null +++ b/messagingobs/messagingobs.go @@ -0,0 +1,331 @@ +package messagingobs + +import ( + "context" + "reflect" + "time" + + constant "github.com/LerianStudio/lib-observability/v2/constants" + "github.com/LerianStudio/lib-observability/v2/tracing" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + "go.opentelemetry.io/otel/trace" +) + +// Metric names (OpenTelemetry messaging semantic conventions). Both are Float64 +// seconds histograms, shared with lib-streaming so worker dashboards are +// transport-agnostic. +const ( + messagingClientOperationDurationMetric = "messaging.client.operation.duration" + messagingProcessDurationMetric = "messaging.process.duration" +) + +// messagingSystemRabbitMQ is the messaging.system attribute value. +const messagingSystemRabbitMQ = "rabbitmq" + +// messagingDurationBuckets follows the HTTP/RPC/Messaging advisory bucket layout +// from docs/metrics-contract.md. +var messagingDurationBuckets = []float64{ + 0.005, 0.01, 0.025, 0.05, 0.075, + 0.1, 0.25, 0.5, 0.75, + 1, 2.5, 5, 7.5, 10, +} + +// newDurationHistogram builds a Float64 seconds histogram for the given metric +// name. Returns nil if the meter is nil or creation fails; callers treat nil as +// "do not record". +func newDurationHistogram(meter metric.Meter, name, description string) metric.Float64Histogram { + if meter == nil { + return nil + } + + hist, err := meter.Float64Histogram( + name, + metric.WithUnit("s"), + metric.WithDescription(description), + metric.WithExplicitBucketBoundaries(messagingDurationBuckets...), + ) + if err != nil { + return nil + } + + return hist +} + +// telemetryEnabled reports whether the Telemetry has the components required to +// emit metrics, mirroring the middleware gate (MeterProvider AND MetricsFactory +// non-nil). +func telemetryEnabled(tl *tracing.Telemetry) bool { + return tl != nil && tl.MeterProvider != nil && tl.MetricsFactory != nil +} + +// tracerFor returns the library tracer when a TracerProvider is configured, or +// nil otherwise (callers create no span for nil). +func tracerFor(tl *tracing.Telemetry) trace.Tracer { + if tl == nil || tl.TracerProvider == nil { + return nil + } + + return tl.TracerProvider.Tracer(tl.LibraryName) +} + +// classifyErrorType returns a bounded error.type label for a failed messaging +// operation. It uses the Go error type name (a bounded set for a given service), +// never the error message, which could carry unbounded/PII content. +func classifyErrorType(err error) string { + if err == nil { + return "" + } + + t := reflect.TypeOf(err) + if t == nil { + return "error" + } + + if t.Kind() == reflect.Pointer { + t = t.Elem() + } + + name := t.String() + if name == "" { + return "error" + } + + return constant.SanitizeMetricLabel(name) +} + +// ProduceParams describes a single outbound message for instrumentation. +type ProduceParams struct { + // DestinationTemplate is the LOW-CARDINALITY destination template, e.g. + // "transactions.{tenant}". NEVER the concrete queue name or routing key. + DestinationTemplate string + // OperationName is the messaging.operation.name, e.g. "publish"/"send". + OperationName string + // RoutingKey and MessageID are accepted for the caller's own logging/span + // body use; they are FORBIDDEN as labels and are NOT emitted as attributes. + RoutingKey string + MessageID string +} + +// ConsumeParams describes a single inbound message for instrumentation. +type ConsumeParams struct { + // Headers are the inbound AMQP-style headers (Delivery.Headers) carrying the + // propagated trace context. + Headers map[string]any + // DestinationTemplate is the LOW-CARDINALITY destination template. + DestinationTemplate string + // OperationName is the messaging.operation.name, e.g. "process"/"receive". + OperationName string + // ConsumerGroup is the messaging.consumer.group.name (bounded). + ConsumerGroup string + // RoutingKey and MessageID are accepted for the caller's own use; FORBIDDEN + // as labels and NOT emitted as attributes. + RoutingKey string + MessageID string +} + +// FinishFunc records the operation duration and ends the span. Call it once, +// passing the operation's error (or nil on success). It is always safe to call, +// even for a no-op (nil-telemetry) helper. +type FinishFunc func(err error) + +// Publisher instruments RabbitMQ produce operations. Build one with NewPublisher +// and reuse it; the duration histogram is created once at construction. +type Publisher struct { + tel *tracing.Telemetry + hist metric.Float64Histogram +} + +// NewPublisher returns a Publisher bound to the given Telemetry. The duration +// histogram is built once here (nil when telemetry is disabled), matching the +// instrument-once pattern used across the library. +func NewPublisher(tl *tracing.Telemetry) *Publisher { + p := &Publisher{tel: tl} + + if telemetryEnabled(tl) { + p.hist = newDurationHistogram( + tl.MeterProvider.Meter(tl.LibraryName), + messagingClientOperationDurationMetric, + "Duration of messaging producer operations.", + ) + } + + return p +} + +// Produce starts a producer span, injects the trace context into a fresh header +// map for the caller to attach to the outgoing AMQP Publishing, and returns a +// FinishFunc that records messaging.client.operation.duration and ends the span. +// +// The returned headers are always non-nil so the caller can attach them +// unconditionally. With telemetry disabled the returned headers still carry any +// propagatable context and the FinishFunc is a safe no-op. +func (p *Publisher) Produce(ctx context.Context, params ProduceParams) (context.Context, map[string]any, FinishFunc) { + if ctx == nil { + ctx = context.Background() + } + + var span trace.Span + + if tracer := tracerFor(p.tel); tracer != nil { + ctx, span = tracer.Start(ctx, spanName(params.OperationName, params.DestinationTemplate), + trace.WithSpanKind(trace.SpanKindProducer), + trace.WithAttributes(baseMessagingAttrs(params.OperationName, params.DestinationTemplate)...), + ) + } + + // Inject trace context into the headers the caller will publish. This works + // from the (possibly span-updated) ctx regardless of whether a span was + // created here. + headers := make(map[string]any) + tracing.InjectTraceHeadersIntoQueue(ctx, &headers) + + start := time.Now() + + finish := func(err error) { + recordMessagingDuration(ctx, p.hist, params.OperationName, params.DestinationTemplate, "", start, err) + finalizeSpan(span, err) + } + + return ctx, headers, finish +} + +// Consumer instruments RabbitMQ consume/process operations. Build one with +// NewConsumer and reuse it. +type Consumer struct { + tel *tracing.Telemetry + hist metric.Float64Histogram +} + +// NewConsumer returns a Consumer bound to the given Telemetry. The process +// duration histogram is built once here (nil when telemetry is disabled). +func NewConsumer(tl *tracing.Telemetry) *Consumer { + c := &Consumer{tel: tl} + + if telemetryEnabled(tl) { + c.hist = newDurationHistogram( + tl.MeterProvider.Meter(tl.LibraryName), + messagingProcessDurationMetric, + "Duration of messaging consumer processing.", + ) + } + + return c +} + +// Consume extracts the trace context from the inbound headers (joining the +// producer's trace), starts a consumer span, and returns a FinishFunc that +// records messaging.process.duration and ends the span. +// +// With telemetry disabled the returned context is the extracted context (still +// useful for downstream propagation) and the FinishFunc is a safe no-op. +func (c *Consumer) Consume(ctx context.Context, params ConsumeParams) (context.Context, FinishFunc) { + if ctx == nil { + ctx = context.Background() + } + + // Always extract inbound trace context so downstream work joins the trace, + // even when this service's telemetry is disabled. + ctx = tracing.ExtractTraceContextFromQueueHeaders(ctx, params.Headers) + + var span trace.Span + + if tracer := tracerFor(c.tel); tracer != nil { + attrs := baseMessagingAttrs(params.OperationName, params.DestinationTemplate) + if params.ConsumerGroup != "" { + attrs = append(attrs, attribute.String("messaging.consumer.group.name", + constant.SanitizeMetricLabel(params.ConsumerGroup))) + } + + ctx, span = tracer.Start(ctx, spanName(params.OperationName, params.DestinationTemplate), + trace.WithSpanKind(trace.SpanKindConsumer), + trace.WithAttributes(attrs...), + ) + } + + start := time.Now() + + finish := func(err error) { + recordMessagingDuration(ctx, c.hist, params.OperationName, params.DestinationTemplate, + params.ConsumerGroup, start, err) + finalizeSpan(span, err) + } + + return ctx, finish +} + +// baseMessagingAttrs builds the bounded attribute set shared by span and metric: +// messaging.system, messaging.operation.name, messaging.destination.template. +func baseMessagingAttrs(operationName, destinationTemplate string) []attribute.KeyValue { + attrs := make([]attribute.KeyValue, 0, 3) + attrs = append(attrs, attribute.String("messaging.system", messagingSystemRabbitMQ)) + + if operationName != "" { + attrs = append(attrs, attribute.String("messaging.operation.name", + constant.SanitizeMetricLabel(operationName))) + } + + if destinationTemplate != "" { + attrs = append(attrs, attribute.String("messaging.destination.template", + constant.SanitizeMetricLabel(destinationTemplate))) + } + + return attrs +} + +// recordMessagingDuration emits a messaging duration observation. It is a no-op +// when the histogram is nil (telemetry disabled or creation failed). The label +// set carries ONLY bounded values; routing key and message id are never added. +func recordMessagingDuration( + ctx context.Context, + hist metric.Float64Histogram, + operationName, destinationTemplate, consumerGroup string, + start time.Time, + err error, +) { + if hist == nil { + return + } + + attrs := baseMessagingAttrs(operationName, destinationTemplate) + + if consumerGroup != "" { + attrs = append(attrs, attribute.String("messaging.consumer.group.name", + constant.SanitizeMetricLabel(consumerGroup))) + } + + if errType := classifyErrorType(err); errType != "" { + attrs = append(attrs, attribute.String("error.type", errType)) + } + + hist.Record(ctx, time.Since(start).Seconds(), metric.WithAttributes(attrs...)) +} + +// finalizeSpan records the error on the span (if any) and ends it. Safe on a nil +// span. +func finalizeSpan(span trace.Span, err error) { + if span == nil { + return + } + + if err != nil { + tracing.HandleSpanError(span, "messaging operation failed", err) + } + + span.End() +} + +// spanName builds an OTel-convention messaging span name "{operation} {template}" +// (both low-cardinality), falling back to whichever part is present. +func spanName(operationName, destinationTemplate string) string { + switch { + case operationName != "" && destinationTemplate != "": + return operationName + " " + destinationTemplate + case destinationTemplate != "": + return destinationTemplate + case operationName != "": + return operationName + default: + return messagingSystemRabbitMQ + } +} diff --git a/messagingobs/messagingobs_test.go b/messagingobs/messagingobs_test.go new file mode 100644 index 0000000..5f5171c --- /dev/null +++ b/messagingobs/messagingobs_test.go @@ -0,0 +1,312 @@ +//go:build unit + +package messagingobs + +import ( + "context" + "errors" + "strings" + "testing" + + "github.com/LerianStudio/lib-observability/v2/metrics" + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/propagation" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" +) + +func newHarness(t *testing.T) (*tracing.Telemetry, *sdkmetric.ManualReader, *tracetest.InMemoryExporter) { + t.Helper() + + // Set the global text-map propagator exactly as the library bootstrap does + // (tracing.otel.go). The queue trace helpers rely on + // otel.GetTextMapPropagator(); without this it is the no-op propagator and no + // headers are injected — the same behavior the app would see if it never + // called InitializeGlobalTelemetry. + otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator( + propagation.TraceContext{}, propagation.Baggage{}, + )) + + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + factory, err := metrics.NewMetricsFactory(mp.Meter("test-library"), nil) + require.NoError(t, err) + + spanExp := tracetest.NewInMemoryExporter() + tp := sdktrace.NewTracerProvider(sdktrace.WithSyncer(spanExp)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + + tel := &tracing.Telemetry{ + TelemetryConfig: tracing.TelemetryConfig{ + LibraryName: "test-library", + EnableTelemetry: true, + }, + TracerProvider: tp, + MeterProvider: mp, + MetricsFactory: factory, + } + + return tel, reader, spanExp +} + +func findHistogram(t *testing.T, reader *sdkmetric.ManualReader, name string) []metricdata.HistogramDataPoint[float64] { + t.Helper() + + rm := &metricdata.ResourceMetrics{} + require.NoError(t, reader.Collect(context.Background(), rm)) + + var points []metricdata.HistogramDataPoint[float64] + + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name != name { + continue + } + + require.Equal(t, "s", m.Unit, "%s must be seconds", name) + + h, ok := m.Data.(metricdata.Histogram[float64]) + require.True(t, ok, "expected float64 histogram for %s, got %T", name, m.Data) + points = append(points, h.DataPoints...) + } + } + + return points +} + +func attrString(set attribute.Set, key string) (string, bool) { + v, ok := set.Value(attribute.Key(key)) + if !ok { + return "", false + } + + return v.AsString(), true +} + +// TestProduce_RecordsDurationInjectsTraceAndOmitsForbidden verifies the producer +// helper: records messaging.client.operation.duration (seconds) with the +// low-cardinality labels, injects trace context into the returned AMQP headers, +// and never emits routing key / message id as a label. +func TestProduce_RecordsDurationInjectsTraceAndOmitsForbidden(t *testing.T) { + tel, reader, spanExp := newHarness(t) + + pub := NewPublisher(tel) + + ctx, headers, finish := pub.Produce(context.Background(), ProduceParams{ + DestinationTemplate: "transactions.{tenant}", + OperationName: "publish", + // These MUST NOT become labels: + RoutingKey: "transactions.acme.pix.9f3c", + MessageID: "0f8fad5b-d9cb-469f-a165-70867728950e", + }) + require.NotNil(t, ctx) + + // Trace context must be injected into the headers the caller will attach to + // the AMQP publishing. The lib's queue helpers canonicalize the header key + // via textproto (e.g. "Traceparent"), so match case-insensitively. + require.NotEmpty(t, headers, "producer must inject trace headers") + assert.True(t, hasTraceparent(headers), "traceparent must be injected for propagation, got %v", headers) + + finish(nil) + + points := findHistogram(t, reader, messagingClientOperationDurationMetric) + require.Len(t, points, 1, "exactly one produce observation expected") + + dp := points[0] + + system, ok := attrString(dp.Attributes, "messaging.system") + require.True(t, ok) + assert.Equal(t, "rabbitmq", system) + + op, ok := attrString(dp.Attributes, "messaging.operation.name") + require.True(t, ok) + assert.Equal(t, "publish", op) + + dest, ok := attrString(dp.Attributes, "messaging.destination.template") + require.True(t, ok) + assert.Equal(t, "transactions.{tenant}", dest) + + // FORBIDDEN labels (docs/metrics-contract.md): routing key & message id. + assertNoForbiddenMessagingLabels(t, dp.Attributes) + + // The span must also exist and be free of forbidden attributes. + spans := spanExp.GetSpans() + require.NotEmpty(t, spans) + + for _, kv := range spans[0].Attributes { + val := kv.Value.Emit() + assert.NotContains(t, val, "transactions.acme.pix.9f3c", "span leaked routing key: %s", kv.Key) + assert.NotContains(t, val, "0f8fad5b", "span leaked message id: %s", kv.Key) + } +} + +// TestConsume_ExtractsTraceRecordsProcessDuration verifies the consumer helper: +// extracts trace context from inbound headers (joining the producer's trace), +// records messaging.process.duration (seconds), and carries the consumer group. +func TestConsume_ExtractsTraceRecordsProcessDuration(t *testing.T) { + tel, reader, spanExp := newHarness(t) + + pub := NewPublisher(tel) + + // Produce first to obtain headers carrying a trace context. + _, headers, finishProduce := pub.Produce(context.Background(), ProduceParams{ + DestinationTemplate: "transactions.{tenant}", + OperationName: "publish", + }) + finishProduce(nil) + + require.True(t, hasTraceparent(headers), "produce must inject traceparent, got %v", headers) + + con := NewConsumer(tel) + + ctx, finish := con.Consume(context.Background(), ConsumeParams{ + Headers: headers, + DestinationTemplate: "transactions.{tenant}", + OperationName: "process", + ConsumerGroup: "ledger-workers", + RoutingKey: "transactions.acme.pix.9f3c", + MessageID: "0f8fad5b-d9cb-469f-a165-70867728950e", + }) + require.NotNil(t, ctx) + + finish(nil) + + points := findHistogram(t, reader, messagingProcessDurationMetric) + require.Len(t, points, 1) + + dp := points[0] + + system, ok := attrString(dp.Attributes, "messaging.system") + require.True(t, ok) + assert.Equal(t, "rabbitmq", system) + + op, ok := attrString(dp.Attributes, "messaging.operation.name") + require.True(t, ok) + assert.Equal(t, "process", op) + + group, ok := attrString(dp.Attributes, "messaging.consumer.group.name") + require.True(t, ok) + assert.Equal(t, "ledger-workers", group) + + assertNoForbiddenMessagingLabels(t, dp.Attributes) + + // The consumer span must be linked to the producer's trace (same trace id). + spans := spanExp.GetSpans() + require.NotEmpty(t, spans) + + var producerTraceID, consumerTraceID string + for _, s := range spans { + if s.SpanKind.String() == "producer" { + producerTraceID = s.SpanContext.TraceID().String() + } + + if s.SpanKind.String() == "consumer" { + consumerTraceID = s.SpanContext.TraceID().String() + } + } + + require.NotEmpty(t, producerTraceID) + require.NotEmpty(t, consumerTraceID) + assert.Equal(t, producerTraceID, consumerTraceID, + "consumer span must join the producer's trace via header propagation") +} + +// TestProduce_ErrorSetsErrorTypeLabel verifies a failed produce records the +// bounded error.type label. +func TestProduce_ErrorSetsErrorTypeLabel(t *testing.T) { + tel, reader, _ := newHarness(t) + + pub := NewPublisher(tel) + + _, _, finish := pub.Produce(context.Background(), ProduceParams{ + DestinationTemplate: "transactions.{tenant}", + OperationName: "publish", + }) + + finish(errors.New("broker unreachable")) + + points := findHistogram(t, reader, messagingClientOperationDurationMetric) + require.Len(t, points, 1) + + errType, ok := attrString(points[0].Attributes, "error.type") + require.True(t, ok, "error.type must be set on a failed produce") + assert.NotEmpty(t, errType) +} + +// TestProduce_NilTelemetryIsNoOp verifies the producer degrades to a safe no-op +// that still returns usable trace headers and never panics. +func TestProduce_NilTelemetryIsNoOp(t *testing.T) { + pub := NewPublisher(nil) + + ctx, headers, finish := pub.Produce(context.Background(), ProduceParams{ + DestinationTemplate: "x", + OperationName: "publish", + }) + require.NotNil(t, ctx) + require.NotNil(t, headers) + + assert.NotPanics(t, func() { finish(nil) }) +} + +// TestConsume_NilTelemetryIsNoOp verifies the consumer degrades to a safe no-op. +func TestConsume_NilTelemetryIsNoOp(t *testing.T) { + con := NewConsumer(nil) + + ctx, finish := con.Consume(context.Background(), ConsumeParams{ + Headers: map[string]any{}, + DestinationTemplate: "x", + OperationName: "process", + }) + require.NotNil(t, ctx) + + assert.NotPanics(t, func() { finish(nil) }) +} + +// hasTraceparent reports whether the AMQP-style header map carries a W3C +// traceparent under any case variant of the key. +func hasTraceparent(headers map[string]any) bool { + for k := range headers { + if strings.EqualFold(k, "traceparent") { + return true + } + } + + return false +} + +// assertNoForbiddenMessagingLabels asserts none of the FORBIDDEN messaging +// labels (routing key, message id, and their common attribute keys) appear on +// the metric. +func assertNoForbiddenMessagingLabels(t *testing.T, set attribute.Set) { + t.Helper() + + forbiddenKeys := []string{ + "messaging.rabbitmq.destination.routing_key", + "messaging.message.id", + "routing_key", + "message_id", + "messaging.destination.name", // concrete queue/routing name (unbounded) + } + + for _, k := range forbiddenKeys { + _, ok := set.Value(attribute.Key(k)) + assert.False(t, ok, "forbidden messaging label %q must not be present", k) + } + + // Also verify no attribute VALUE contains the concrete routing key / id. + for _, kv := range set.ToSlice() { + val := kv.Value.Emit() + assert.NotContains(t, val, "transactions.acme.pix.9f3c", + "metric label %q leaked routing key: %s", kv.Key, val) + assert.NotContains(t, val, "0f8fad5b", + "metric label %q leaked message id: %s", kv.Key, val) + } +} diff --git a/middleware/helpers.go b/middleware/helpers.go index 661f40b..4425ec1 100644 --- a/middleware/helpers.go +++ b/middleware/helpers.go @@ -191,26 +191,6 @@ func sanitizeLogValue(raw string) string { return replacer.Replace(raw) } -// getGRPCUserAgent extracts the User-Agent from incoming gRPC metadata. -// Returns empty string if the metadata is not present or doesn't contain user-agent. -func getGRPCUserAgent(ctx context.Context) string { - if ctx == nil { - return "" - } - - md, ok := metadata.FromIncomingContext(ctx) - if !ok || md == nil { - return "" - } - - userAgents := md.Get(strings.ToLower(headerUserAgent)) - if len(userAgents) == 0 { - return "" - } - - return userAgents[0] -} - // isInternalLerianService reports whether a user-agent belongs to a Lerian internal service. func isInternalLerianService(userAgent string) bool { return internalServicePattern.MatchString(userAgent) diff --git a/middleware/http_trace.go b/middleware/http_trace.go new file mode 100644 index 0000000..460b36b --- /dev/null +++ b/middleware/http_trace.go @@ -0,0 +1,55 @@ +package middleware + +import ( + "context" + + observability "github.com/LerianStudio/lib-observability/v2" + "github.com/LerianStudio/lib-observability/v2/redaction" + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/gofiber/fiber/v3" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/propagation" +) + +// SetSpanAttributeForParam adds a request parameter attribute to the current context bag. +// Sensitive parameter names (as determined by redaction.IsSensitiveField) are masked. +// +// Moved here from the tracing package: it depends on fiber.Ctx, which must not +// leak into the tracing core so Fiber-v2 apps can consume tracing without pulling +// in Fiber v3. Behavior is identical to the previous tracing.SetSpanAttributeForParam. +func SetSpanAttributeForParam(c fiber.Ctx, param, value, entityName string) { + if c == nil { + return + } + + spanAttrKey := "app.request." + param + if entityName != "" && param == "id" { + spanAttrKey = "app.request." + entityName + "_id" + } + + // Mask value if the parameter name is considered sensitive + attrValue := value + if redaction.IsSensitiveField(param) { + attrValue = "[REDACTED]" + } + + c.SetContext(observability.ContextWithSpanAttributes(c.Context(), attribute.String(spanAttrKey, attrValue))) +} + +// ExtractHTTPContext extracts trace headers from a Fiber request. +// +// Moved here from the tracing package: it depends on fiber.Ctx. It delegates the +// carrier-level extraction to tracing.ExtractTraceContext, which remains +// fiber-free. Behavior is identical to the previous tracing.ExtractHTTPContext. +func ExtractHTTPContext(ctx context.Context, c fiber.Ctx) context.Context { + if c == nil { + return ctx + } + + carrier := propagation.HeaderCarrier{} + for key, value := range c.Request().Header.All() { + carrier.Set(string(key), string(value)) + } + + return tracing.ExtractTraceContext(ctx, carrier) +} diff --git a/middleware/metrics.go b/middleware/metrics.go index 48dfdfc..f086bb2 100644 --- a/middleware/metrics.go +++ b/middleware/metrics.go @@ -2,135 +2,27 @@ package middleware import ( "context" - "errors" - "os" - "sync" - "time" - observability "github.com/LerianStudio/lib-observability/v2" - "github.com/LerianStudio/lib-observability/v2/runtime" + "github.com/LerianStudio/lib-observability/v2/telemetrycore" ) -// DefaultMetricsCollectionInterval is the default interval for collecting system metrics. -// Can be overridden via METRICS_COLLECTION_INTERVAL environment variable. -const DefaultMetricsCollectionInterval = 5 * time.Second +// DefaultMetricsCollectionInterval is re-exported from telemetrycore for +// backward compatibility with callers of the middleware package. +const DefaultMetricsCollectionInterval = telemetrycore.DefaultMetricsCollectionInterval -// Metrics collector singleton state. -var ( - metricsCollectorOnce = &sync.Once{} - metricsCollectorShutdown chan struct{} - metricsCollectorMu sync.Mutex - metricsCollectorStarted bool - metricsCollectorInitErr error -) - -// telemetryRuntimeLogger returns the runtime logger from the telemetry middleware, or nil. -func telemetryRuntimeLogger(tm *TelemetryMiddleware) runtime.Logger { - if tm == nil || tm.Telemetry == nil { - return nil - } - - return tm.Telemetry.Logger +// StopMetricsCollector stops the background metrics collector goroutine. +// It delegates to telemetrycore so the HTTP middleware and the gRPC +// interceptors share a single collector singleton. Re-exported here for +// backward compatibility with callers of the middleware package. +func StopMetricsCollector() { + telemetrycore.StopMetricsCollector() } // collectMetrics ensures the background metrics collector goroutine is running. func (tm *TelemetryMiddleware) collectMetrics(_ context.Context) error { - return tm.ensureMetricsCollector() -} - -// getMetricsCollectionInterval returns the metrics collection interval. -// Can be configured via METRICS_COLLECTION_INTERVAL environment variable. -// Accepts Go duration format (e.g., "10s", "1m", "500ms"). -// Falls back to DefaultMetricsCollectionInterval if not set or invalid. -func getMetricsCollectionInterval() time.Duration { - if envInterval := os.Getenv("METRICS_COLLECTION_INTERVAL"); envInterval != "" { - if parsed, err := time.ParseDuration(envInterval); err == nil && parsed > 0 { - return parsed - } - } - - return DefaultMetricsCollectionInterval -} - -// ensureMetricsCollector lazily starts the background metrics collector singleton. -func (tm *TelemetryMiddleware) ensureMetricsCollector() error { - if tm == nil || tm.Telemetry == nil { - return nil - } - - if tm.Telemetry.MeterProvider == nil { - return nil - } - - metricsCollectorMu.Lock() - defer metricsCollectorMu.Unlock() - - if metricsCollectorStarted { + if tm == nil { return nil } - if metricsCollectorInitErr != nil { - metricsCollectorOnce = &sync.Once{} - metricsCollectorInitErr = nil - } - - metricsCollectorOnce.Do(func() { - factory := tm.Telemetry.MetricsFactory - if factory == nil { - metricsCollectorInitErr = errors.New("telemetry MetricsFactory is nil, cannot start system metrics collector") - return - } - - shutdown := make(chan struct{}) - metricsCollectorShutdown = shutdown - ticker := time.NewTicker(getMetricsCollectionInterval()) - - runtime.SafeGoWithContextAndComponent( - context.Background(), - telemetryRuntimeLogger(tm), - "http", - "metrics_collector", - runtime.KeepRunning, - func(_ context.Context) { - observability.GetCPUUsage(context.Background(), factory) - observability.GetMemUsage(context.Background(), factory) - - for { - select { - case <-shutdown: - ticker.Stop() - return - case <-ticker.C: - observability.GetCPUUsage(context.Background(), factory) - observability.GetMemUsage(context.Background(), factory) - } - } - }, - ) - - metricsCollectorStarted = true - }) - - return metricsCollectorInitErr -} - -// StopMetricsCollector stops the background metrics collector goroutine. -// Should be called during application shutdown for graceful cleanup. -// After calling this function, the collector can be restarted by new requests. -// -// Implementation note: This function intentionally resets sync.Once to a new instance -// to allow the collector to be restarted after being stopped. This is an unusual but -// intentional pattern - the mutex ensures thread-safety during the reset operation, -// preventing race conditions between Stop and subsequent Start calls. -func StopMetricsCollector() { - metricsCollectorMu.Lock() - defer metricsCollectorMu.Unlock() - - if metricsCollectorStarted && metricsCollectorShutdown != nil { - close(metricsCollectorShutdown) - - metricsCollectorStarted = false - metricsCollectorOnce = &sync.Once{} - metricsCollectorInitErr = nil - } + return telemetrycore.EnsureMetricsCollector(tm.Telemetry) } diff --git a/middleware/telemetry.go b/middleware/telemetry.go index b239433..4d3f977 100644 --- a/middleware/telemetry.go +++ b/middleware/telemetry.go @@ -1,5 +1,10 @@ -// Package middleware provides Fiber HTTP and gRPC telemetry middleware that -// integrates with the lib-observability tracing and metrics packages. +// Package middleware provides Fiber HTTP telemetry middleware that integrates +// with the lib-observability tracing and metrics packages. +// +// The gRPC telemetry interceptors live in the sibling grpcmiddleware package, +// which is Fiber-free so Fiber-v2 applications can consume them without pulling +// in Fiber v3. Both packages share the single process-wide system-metrics +// collector via telemetrycore. package middleware import ( @@ -21,15 +26,17 @@ import ( "go.opentelemetry.io/otel/codes" "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/trace" - "google.golang.org/grpc" - "google.golang.org/grpc/metadata" - "google.golang.org/grpc/status" ) // httpServerRequestDurationMetric is the OpenTelemetry semantic-convention metric name // for HTTP server request duration. Recorded as a Float64 histogram in seconds. const httpServerRequestDurationMetric = "http.server.request.duration" +// httpServerActiveRequestsMetric is the OpenTelemetry semantic-convention metric +// name for the number of in-flight HTTP server requests. Recorded as an Int64 +// UpDownCounter in the unitless "{request}" dimension. +const httpServerActiveRequestsMetric = "http.server.active_requests" + // httpServerDurationBuckets follows the current OpenTelemetry HTTP semantic // conventions advisory layout for http.server.request.duration. Update only // in lockstep with the spec. @@ -60,6 +67,26 @@ func newHTTPServerDurationHistogram(meter metric.Meter) metric.Float64Histogram return hist } +// newActiveRequestsCounter builds the int64 UpDownCounter instrument for +// http.server.active_requests on the given meter. Returns nil if the meter is +// nil or instrument creation fails - callers must treat nil as "do not record". +func newActiveRequestsCounter(meter metric.Meter) metric.Int64UpDownCounter { + if meter == nil { + return nil + } + + counter, err := meter.Int64UpDownCounter( + httpServerActiveRequestsMetric, + metric.WithUnit("{request}"), + metric.WithDescription("Number of active HTTP server requests."), + ) + if err != nil { + return nil + } + + return counter +} + // Header and metadata key constants used by the middleware. const ( // headerID is the request identifier header key. @@ -124,7 +151,7 @@ func spanEndStateFromContext(ctx context.Context) *spanEndState { return state } -// TelemetryMiddleware wraps HTTP and gRPC handlers with tracing and metrics setup. +// TelemetryMiddleware wraps Fiber HTTP handlers with tracing and metrics setup. type TelemetryMiddleware struct { Telemetry *tracing.Telemetry } @@ -149,7 +176,10 @@ func (tm *TelemetryMiddleware) WithTelemetry(tl *tracing.Telemetry, excludedRout // or via the receiver's stored Telemetry, mirroring the per-request logic // below. If neither resolves, or any required component is nil, the // histogram is left nil and recording is skipped. - var durationHistogram metric.Float64Histogram + var ( + durationHistogram metric.Float64Histogram + activeRequests metric.Int64UpDownCounter + ) bootstrapTelemetry := tl if bootstrapTelemetry == nil && tm != nil { @@ -165,9 +195,9 @@ func (tm *TelemetryMiddleware) WithTelemetry(tl *tracing.Telemetry, excludedRout if bootstrapTelemetry != nil && bootstrapTelemetry.MeterProvider != nil && bootstrapTelemetry.MetricsFactory != nil { - durationHistogram = newHTTPServerDurationHistogram( - bootstrapTelemetry.MeterProvider.Meter(bootstrapTelemetry.LibraryName), - ) + meter := bootstrapTelemetry.MeterProvider.Meter(bootstrapTelemetry.LibraryName) + durationHistogram = newHTTPServerDurationHistogram(meter) + activeRequests = newActiveRequestsCounter(meter) } return func(c fiber.Ctx) error { @@ -210,6 +240,13 @@ func (tm *TelemetryMiddleware) WithTelemetry(tl *tracing.Telemetry, excludedRout rawMethod := string([]byte(c.Method())) method, methodOriginal, methodReplaced := normalizeHTTPMethod(rawMethod) + // Track in-flight requests around the full downstream chain. Increment + // before c.Next() and decrement on return via the returned closure, so + // the counter reflects concurrency across both the tracing and + // no-tracer paths below. No-op when the instrument is nil. + activeDone := trackActiveRequest(c.Context(), activeRequests, method) + defer activeDone() + if effectiveTelemetry.TracerProvider == nil { err := c.Next() @@ -238,7 +275,7 @@ func (tm *TelemetryMiddleware) WithTelemetry(tl *tracing.Telemetry, excludedRout // User-Agent heuristic. This is an interoperability hint, not an authenticated // trust boundary, and is preserved to avoid changing existing caller behavior. if isInternalLerianService(userAgent) { - traceCtx = tracing.ExtractHTTPContext(traceCtx, c) + traceCtx = ExtractHTTPContext(traceCtx, c) } ctx, span := tracer.Start(traceCtx, spanName, trace.WithSpanKind(trace.SpanKindServer)) @@ -417,6 +454,27 @@ func recordHTTPServerDuration( hist.Record(c.Context(), durationSeconds, metric.WithAttributes(attrs...)) } +// trackActiveRequest increments the http.server.active_requests UpDownCounter by +// one and returns a closure that decrements it by one when invoked (deferred by +// the caller). The label set is intentionally minimal - only +// http.request.method - to keep the concurrency gauge low-cardinality; +// http.route is deliberately omitted because it is not reliably known before +// routing (c.Next), and adding it would multiply the series without adding +// operational value for an in-flight gauge. It is a no-op (returns a no-op +// closure) when the counter is nil, so callers can invoke it unconditionally. +func trackActiveRequest(ctx context.Context, counter metric.Int64UpDownCounter, method string) func() { + if counter == nil { + return func() {} + } + + attrs := metric.WithAttributes(attribute.String("http.request.method", method)) + counter.Add(ctx, 1, attrs) + + return func() { + counter.Add(ctx, -1, attrs) + } +} + // classifyHTTPErrorType returns the stable, low-cardinality error.type // label for the http.server.request.duration metric per OpenTelemetry HTTP // semantic conventions. Status-driven by design: a 503 surfaced via @@ -469,122 +527,6 @@ func (tm *TelemetryMiddleware) EndTracingSpans(c fiber.Ctx) error { return err } -// WithTelemetryInterceptor is a gRPC interceptor that adds tracing to the context. -func (tm *TelemetryMiddleware) WithTelemetryInterceptor(tl *tracing.Telemetry) grpc.UnaryServerInterceptor { - return func( - ctx context.Context, - req any, - info *grpc.UnaryServerInfo, - handler grpc.UnaryHandler, - ) (any, error) { - ctx = normalizeGRPCContext(ctx) - - effectiveTelemetry := tl - if effectiveTelemetry == nil && tm != nil { - effectiveTelemetry = tm.Telemetry - } - - if effectiveTelemetry == nil { - return handler(ctx, req) - } - - requestID := resolveGRPCRequestID(ctx, req) - ctx = observability.ContextWithHeaderID(ctx, requestID) - - if effectiveTelemetry.TracerProvider == nil { - return handler(ctx, req) - } - - tracer := effectiveTelemetry.TracerProvider.Tracer(effectiveTelemetry.LibraryName) - - methodName := "unknown" - if info != nil { - methodName = info.FullMethod - } - - if tenantID := ResolveTenantIDFromGRPC(ctx); tenantID != "" { - ctx = observability.ContextWithSpanAttributes(ctx, attribute.String(constant.AttrKeyTenantID, tenantID)) - } - - ctx = observability.ContextWithSpanAttributes(ctx, - attribute.String("app.request.request_id", requestID), - attribute.String("grpc.method", methodName), - ) - - traceCtx := ctx - // Compatibility note: trace extraction currently trusts the internal-service - // User-Agent heuristic. This is an interoperability hint, not an authenticated - // trust boundary, and is preserved to avoid changing existing caller behavior. - if isInternalLerianService(getGRPCUserAgent(ctx)) { - md, _ := metadata.FromIncomingContext(ctx) - traceCtx = tracing.ExtractGRPCContext(ctx, md) - } - - ctx, span := tracer.Start(traceCtx, methodName, trace.WithSpanKind(trace.SpanKindServer)) - endState := newSpanEndState(span) - // WithTelemetryInterceptor owns this span's lifecycle: it applies - // rpc.method / rpc.grpc.status_code / handler error status AFTER the - // handler returns (below), then ends the span via the defer. Marking it - // owned makes EndTracingSpansInterceptor skip it, so those post-handler - // attributes can't be dropped by a chain where the end interceptor - // unwinds first — mirroring the HTTP WithTelemetry/EndTracingSpans pair. - endState.owned = true - - defer endState.End() - - ctx = observability.ContextWithTracer(ctx, tracer) - ctx = observability.ContextWithMetricFactory(ctx, effectiveTelemetry.MetricsFactory) - ctx = contextWithSpanEndState(ctx, endState) - - err := tm.collectMetrics(ctx) - if err != nil { - tracing.HandleSpanError(span, "Failed to collect metrics", err) - } - - resp, err := handler(ctx, req) - - grpcStatusCode := status.Code(err) - span.SetAttributes( - attribute.String("rpc.method", methodName), - attribute.Int("rpc.grpc.status_code", int(grpcStatusCode)), - ) - - if err != nil { - tracing.HandleSpanError(span, "gRPC handler error", err) - } - - return resp, err - } -} - -// EndTracingSpansInterceptor is a gRPC interceptor that ends the tracing spans. -func (tm *TelemetryMiddleware) EndTracingSpansInterceptor() grpc.UnaryServerInterceptor { - return func( - ctx context.Context, - req any, - info *grpc.UnaryServerInfo, - handler grpc.UnaryHandler, - ) (any, error) { - resp, err := handler(ctx, req) - if state := spanEndStateFromContext(ctx); state != nil { - // A span owned by WithTelemetryInterceptor is finalized and ended by - // that interceptor after it records post-handler attributes/status. - // Skip it here (same reasoning as the HTTP EndTracingSpans). - if state.owned { - return resp, err - } - - state.End() - - return resp, err - } - - trace.SpanFromContext(ctx).End() - - return resp, err - } -} - // setRequestHeaderID ensures the Fiber request carries a unique correlation ID header. // The effective ID is always echoed back on the response so that callers can // correlate their request regardless of whether the ID was client-supplied or diff --git a/middleware/telemetry_active_requests_test.go b/middleware/telemetry_active_requests_test.go new file mode 100644 index 0000000..afd1414 --- /dev/null +++ b/middleware/telemetry_active_requests_test.go @@ -0,0 +1,155 @@ +//go:build unit + +package middleware + +import ( + "context" + "net/http" + "net/http/httptest" + "testing" + + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/gofiber/fiber/v3" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" +) + +// findActiveRequestsSum extracts the http.server.active_requests UpDownCounter +// value. Returns (value, true) if the metric exists, or (0, false) when absent. +// It also locks the unit to "{request}" per the metric contract. +func findActiveRequestsSum( + t *testing.T, + reader *sdkmetric.ManualReader, +) (int64, bool) { + t.Helper() + + rm := &metricdata.ResourceMetrics{} + require.NoError(t, reader.Collect(context.Background(), rm)) + + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name != httpServerActiveRequestsMetric { + continue + } + + s, ok := m.Data.(metricdata.Sum[int64]) + require.True(t, ok, "expected Int64 sum for %s, got %T", m.Name, m.Data) + require.Equal(t, "{request}", m.Unit, "metric unit must be {request}") + require.NotEmpty(t, s.DataPoints) + + return s.DataPoints[0].Value, true + } + } + + return 0, false +} + +// TestWithTelemetry_ActiveRequestsSettlesToZero verifies that after a request +// completes, the active-requests UpDownCounter has been incremented and then +// decremented back to a net zero, and carries the http.request.method label. +func TestWithTelemetry_ActiveRequestsSettlesToZero(t *testing.T) { + tel, reader := newMetricsHarness(t) + + app := fiber.New() + mid := NewTelemetryMiddleware(tel) + app.Use(mid.WithTelemetry(tel)) + + app.Get("/api/ping", func(c fiber.Ctx) error { + return c.SendStatus(http.StatusOK) + }) + + resp, err := app.Test(httptest.NewRequest(http.MethodGet, "/api/ping", nil)) + require.NoError(t, err) + defer func() { require.NoError(t, resp.Body.Close()) }() + assert.Equal(t, http.StatusOK, resp.StatusCode) + + value, present := findActiveRequestsSum(t, reader) + require.True(t, present, "http.server.active_requests must be registered after a request") + assert.EqualValues(t, 0, value, + "active requests must settle back to zero after the request completes") +} + +// TestWithTelemetry_ActiveRequestsIncrementsDuringHandler verifies the counter +// reads exactly 1 while a handler is mid-flight, proving increment happens +// before c.Next() and decrement after. +func TestWithTelemetry_ActiveRequestsIncrementsDuringHandler(t *testing.T) { + tel, reader := newMetricsHarness(t) + + app := fiber.New() + mid := NewTelemetryMiddleware(tel) + app.Use(mid.WithTelemetry(tel)) + + var inFlight int64 + + var methodDuringHandler string + + var hasMethodLabel bool + + app.Get("/api/ping", func(c fiber.Ctx) error { + // Collect while still inside the handler: the counter must read 1. + rm := &metricdata.ResourceMetrics{} + require.NoError(t, reader.Collect(context.Background(), rm)) + + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name != httpServerActiveRequestsMetric { + continue + } + + s, ok := m.Data.(metricdata.Sum[int64]) + require.True(t, ok) + require.NotEmpty(t, s.DataPoints) + inFlight = s.DataPoints[0].Value + + mv, present := s.DataPoints[0].Attributes.Value(attribute.Key("http.request.method")) + hasMethodLabel = present + methodDuringHandler = mv.AsString() + } + } + + return c.SendStatus(http.StatusOK) + }) + + resp, err := app.Test(httptest.NewRequest(http.MethodGet, "/api/ping", nil)) + require.NoError(t, err) + defer func() { require.NoError(t, resp.Body.Close()) }() + + assert.EqualValues(t, 1, inFlight, + "active requests must read 1 while the handler is executing") + require.True(t, hasMethodLabel, "active requests must carry http.request.method") + assert.Equal(t, "GET", methodDuringHandler) +} + +// TestWithTelemetry_ActiveRequestsNilMetricsFactoryDoesNotRecord verifies the +// active-requests counter is gated on MetricsFactory presence, matching the +// duration histogram. +func TestWithTelemetry_ActiveRequestsNilMetricsFactoryDoesNotRecord(t *testing.T) { + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + tel := &tracing.Telemetry{ + TelemetryConfig: tracing.TelemetryConfig{LibraryName: "test-library"}, + MeterProvider: mp, + // MetricsFactory intentionally nil. + } + + app := fiber.New() + mid := NewTelemetryMiddleware(tel) + app.Use(mid.WithTelemetry(tel)) + + app.Get("/api/ping", func(c fiber.Ctx) error { + return c.SendStatus(http.StatusOK) + }) + + resp, err := app.Test(httptest.NewRequest(http.MethodGet, "/api/ping", nil)) + require.NoError(t, err) + defer func() { require.NoError(t, resp.Body.Close()) }() + + _, present := findActiveRequestsSum(t, reader) + assert.False(t, present, + "nil MetricsFactory must not register http.server.active_requests") +} diff --git a/middleware/telemetry_test.go b/middleware/telemetry_test.go index 9c68da7..75a0acb 100644 --- a/middleware/telemetry_test.go +++ b/middleware/telemetry_test.go @@ -7,23 +7,18 @@ import ( "errors" "net/http" "net/http/httptest" - "sync" "testing" "time" - "github.com/LerianStudio/lib-observability/v2/metrics" "github.com/LerianStudio/lib-observability/v2/tracing" "github.com/gofiber/fiber/v3" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/propagation" - sdkmetric "go.opentelemetry.io/otel/sdk/metric" sdktrace "go.opentelemetry.io/otel/sdk/trace" "go.opentelemetry.io/otel/sdk/trace/tracetest" oteltrace "go.opentelemetry.io/otel/trace" - "google.golang.org/grpc" - "google.golang.org/grpc/metadata" ) // setupTestTracer sets up a test tracer provider and returns it along with a span recorder. @@ -445,123 +440,6 @@ func TestEndTracingSpans_EndsFinalContextSpan(t *testing.T) { assert.Equal(t, "handler-span", spanRecorder.Ended()[0].Name()) } -// TestGetMetricsCollectionInterval tests the getMetricsCollectionInterval function. -func TestGetMetricsCollectionInterval(t *testing.T) { - tests := []struct { - name string - envValue string - expected time.Duration - }{ - { - name: "default when not set", - envValue: "", - expected: DefaultMetricsCollectionInterval, - }, - { - name: "valid duration in seconds", - envValue: "10s", - expected: 10 * time.Second, - }, - { - name: "valid duration in milliseconds", - envValue: "500ms", - expected: 500 * time.Millisecond, - }, - { - name: "valid duration in minutes", - envValue: "1m", - expected: 1 * time.Minute, - }, - { - name: "invalid format falls back to default", - envValue: "invalid", - expected: DefaultMetricsCollectionInterval, - }, - { - name: "zero value falls back to default", - envValue: "0s", - expected: DefaultMetricsCollectionInterval, - }, - { - name: "negative value falls back to default", - envValue: "-5s", - expected: DefaultMetricsCollectionInterval, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - if tt.envValue != "" { - t.Setenv("METRICS_COLLECTION_INTERVAL", tt.envValue) - } else { - t.Setenv("METRICS_COLLECTION_INTERVAL", "") - } - - result := getMetricsCollectionInterval() - assert.Equal(t, tt.expected, result) - }) - } -} - -func resetMetricsCollectorState() { - metricsCollectorMu.Lock() - defer metricsCollectorMu.Unlock() - - if metricsCollectorStarted && metricsCollectorShutdown != nil { - close(metricsCollectorShutdown) - time.Sleep(50 * time.Millisecond) - } - - metricsCollectorShutdown = nil - metricsCollectorStarted = false - metricsCollectorOnce = &sync.Once{} - metricsCollectorInitErr = nil -} - -func TestEnsureMetricsCollector_ReturnsErrorWhenMetricsFactoryNil(t *testing.T) { - resetMetricsCollectorState() - t.Cleanup(resetMetricsCollectorState) - - mid := &TelemetryMiddleware{Telemetry: &tracing.Telemetry{ - TelemetryConfig: tracing.TelemetryConfig{LibraryName: "test-library", EnableTelemetry: true}, - MeterProvider: sdkmetric.NewMeterProvider(), - }} - - err := mid.ensureMetricsCollector() - require.Error(t, err) - assert.Contains(t, err.Error(), "MetricsFactory is nil") - assert.False(t, metricsCollectorStarted) -} - -func TestEnsureMetricsCollector_NoMeterProviderReturnsNil(t *testing.T) { - resetMetricsCollectorState() - t.Cleanup(resetMetricsCollectorState) - - mid := &TelemetryMiddleware{Telemetry: &tracing.Telemetry{}} - require.NoError(t, mid.ensureMetricsCollector()) - assert.False(t, metricsCollectorStarted) -} - -func TestStopMetricsCollector_AllowsRestart(t *testing.T) { - resetMetricsCollectorState() - t.Cleanup(resetMetricsCollectorState) - - mid := &TelemetryMiddleware{Telemetry: &tracing.Telemetry{ - TelemetryConfig: tracing.TelemetryConfig{LibraryName: "test-library", EnableTelemetry: true}, - MeterProvider: sdkmetric.NewMeterProvider(), - MetricsFactory: metrics.NewNopFactory(), - }} - - require.NoError(t, mid.ensureMetricsCollector()) - assert.True(t, metricsCollectorStarted) - - StopMetricsCollector() - assert.False(t, metricsCollectorStarted) - - require.NoError(t, mid.ensureMetricsCollector()) - assert.True(t, metricsCollectorStarted) -} - // TestExtractHTTPContext tests the ExtractHTTPContext function from tracing package. func TestExtractHTTPContext(t *testing.T) { ctx := context.Background() @@ -580,7 +458,7 @@ func TestExtractHTTPContext(t *testing.T) { app := fiber.New() app.Get("/test", func(c fiber.Ctx) error { - ctx := tracing.ExtractHTTPContext(c.Context(), c) + ctx := ExtractHTTPContext(c.Context(), c) spanCtx := oteltrace.SpanContextFromContext(ctx) @@ -705,51 +583,6 @@ func TestWithTelemetryConditionalTracePropagation(t *testing.T) { } } -// TestGetGRPCUserAgent tests the getGRPCUserAgent helper function. -func TestGetGRPCUserAgent(t *testing.T) { - tests := []struct { - name string - setupMetadata func() context.Context - expectedUA string - description string - }{ - { - name: "Valid user-agent in metadata", - setupMetadata: func() context.Context { - md := metadata.Pairs("user-agent", "midaz/1.0.0 LerianStudio") - return metadata.NewIncomingContext(context.Background(), md) - }, - expectedUA: "midaz/1.0.0 LerianStudio", - description: "Should extract user-agent from gRPC metadata", - }, - { - name: "No metadata in context", - setupMetadata: func() context.Context { - return context.Background() - }, - expectedUA: "", - description: "Should return empty string when no metadata present", - }, - { - name: "Metadata without user-agent", - setupMetadata: func() context.Context { - md := metadata.Pairs("authorization", "Bearer token") - return metadata.NewIncomingContext(context.Background(), md) - }, - expectedUA: "", - description: "Should return empty string when user-agent key not present", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - ctx := tt.setupMetadata() - result := getGRPCUserAgent(ctx) - assert.Equal(t, tt.expectedUA, result, tt.description) - }) - } -} - // --------------------------------------------------------------------------- // sanitizeURL tests // --------------------------------------------------------------------------- @@ -814,90 +647,3 @@ func TestSanitizeURL_RelativePath(t *testing.T) { result := sanitizeURL("/api/v1/users?token=abc123") assert.NotContains(t, result, "abc123") } - -// TestWithTelemetryInterceptorConditionalTracePropagation tests conditional trace propagation in gRPC interceptor. -func TestWithTelemetryInterceptorConditionalTracePropagation(t *testing.T) { - tests := []struct { - name string - userAgent string - traceparent string - shouldPropagateTrace bool - description string - }{ - { - name: "Internal Lerian service via gRPC - should propagate trace", - userAgent: "midaz/1.0.0 LerianStudio", - traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", - shouldPropagateTrace: true, - description: "Internal gRPC service should propagate trace context", - }, - { - name: "External gRPC client - should NOT propagate trace", - userAgent: "grpc-go/1.50.0", - traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", - shouldPropagateTrace: false, - description: "External gRPC client should create new root span", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - ctx := context.Background() - - tp, spanRecorder := setupTestTracer(t) - defer func() { - _ = tp.Shutdown(ctx) - }() - - oldTracerProvider := otel.GetTracerProvider() - otel.SetTracerProvider(tp) - defer otel.SetTracerProvider(oldTracerProvider) - - tel := &tracing.Telemetry{ - TelemetryConfig: tracing.TelemetryConfig{ - LibraryName: "test-library", - EnableTelemetry: true, - }, - TracerProvider: tp, - } - - mid := NewTelemetryMiddleware(tel) - interceptor := mid.WithTelemetryInterceptor(tel) - - md := metadata.New(map[string]string{}) - if tt.userAgent != "" { - md.Set("user-agent", tt.userAgent) - } - if tt.traceparent != "" { - md.Set("traceparent", tt.traceparent) - } - ctx = metadata.NewIncomingContext(ctx, md) - - var capturedSpanContext oteltrace.SpanContext - handler := func(ctx context.Context, req any) (any, error) { - capturedSpanContext = oteltrace.SpanContextFromContext(ctx) - return "response", nil - } - - info := &grpc.UnaryServerInfo{ - FullMethod: "/test.Service/Method", - } - - _, err := interceptor(ctx, "request", info, handler) - require.NoError(t, err) - - spans := spanRecorder.Ended() - require.GreaterOrEqual(t, len(spans), 1, "Expected at least one span to be created") - - if tt.shouldPropagateTrace { - assert.True(t, capturedSpanContext.IsValid(), "Span context should be valid for internal services") - assert.Equal(t, "4bf92f3577b34da6a3ce929d0e0e4736", capturedSpanContext.TraceID().String(), - "Trace ID should match the traceparent for internal gRPC services") - } else { - require.True(t, capturedSpanContext.IsValid(), "Expected middleware to attach a valid span context") - assert.NotEqual(t, "4bf92f3577b34da6a3ce929d0e0e4736", capturedSpanContext.TraceID().String(), - "Trace ID should be different from traceparent for external services") - } - }) - } -} diff --git a/redisobs/doc.go b/redisobs/doc.go new file mode 100644 index 0000000..58754cc --- /dev/null +++ b/redisobs/doc.go @@ -0,0 +1,30 @@ +// Package redisobs provides a thin, nil-safe helper that adds OpenTelemetry +// tracing and metrics to a go-redis client the application already created. It +// covers both Redis AND Valkey: Valkey is wire-compatible with Redis and uses +// the same github.com/redis/go-redis/v9 driver, so redisotel instruments both +// unchanged (ADR-004) — the emitted db.system value is "redis" in both cases. +// +// # Boundary (ADR-007) +// +// This package does NOT create or own the client. The application builds its +// redis.UniversalClient (single-node, cluster, or failover) and passes it here; +// the helper applies redisotel.InstrumentTracing + redisotel.InstrumentMetrics +// and returns. It never dials, never closes, and never manages the client. +// +// # Emitted telemetry +// +// redisotel emits db.client.operation.duration (seconds) and command spans with +// db.system=redis. No connection is created by this package. +// +// # PII / cardinality guardrail (docs/metrics-contract.md) +// +// redisotel attaches db.statement — the raw command including the key and, for +// writes, argument values — to spans by default. This package disables that +// unconditionally (WithDBStatement(false)), so no redis key, value, or command +// text is ever emitted as a span or metric attribute. Enforced by tests. +// +// # No-op degradation (ADR-008) +// +// With no providers supplied, instrumentation attaches against the OTel no-op +// providers; the helper never panics and never breaks the client. +package redisobs diff --git a/redisobs/redisobs.go b/redisobs/redisobs.go new file mode 100644 index 0000000..a846edf --- /dev/null +++ b/redisobs/redisobs.go @@ -0,0 +1,122 @@ +package redisobs + +import ( + "errors" + + constant "github.com/LerianStudio/lib-observability/v2/constants" + "github.com/redis/go-redis/extra/redisotel/v9" + "github.com/redis/go-redis/v9" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + "go.opentelemetry.io/otel/trace" +) + +// ErrNilClient is returned by Instrument when the supplied client is nil. +var ErrNilClient = errors.New("redisobs: nil redis.UniversalClient") + +// config holds resolved helper options. +type config struct { + meterProvider metric.MeterProvider + tracerProvider trace.TracerProvider + extraAttrs []attribute.KeyValue +} + +// Option configures the redis instrumentation helper. +type Option func(*config) + +// WithMeterProvider sets the MeterProvider for db.client.operation.duration. +// When unset the global provider is used (no-op unless configured). +func WithMeterProvider(mp metric.MeterProvider) Option { + return func(c *config) { + if mp != nil { + c.meterProvider = mp + } + } +} + +// WithTracerProvider sets the TracerProvider for redis command spans. When unset +// the global provider is used. +func WithTracerProvider(tp trace.TracerProvider) Option { + return func(c *config) { + if tp != nil { + c.tracerProvider = tp + } + } +} + +// WithAttributes appends additional low-cardinality, PII-free attributes to the +// redis spans and metrics. Keys, values, and command text are FORBIDDEN +// (docs/metrics-contract.md) and are never added by this helper. +func WithAttributes(attrs ...attribute.KeyValue) Option { + return func(c *config) { + c.extraAttrs = append(c.extraAttrs, attrs...) + } +} + +func newConfig(opts ...Option) config { + cfg := config{ + meterProvider: otel.GetMeterProvider(), + tracerProvider: otel.GetTracerProvider(), + } + + for _, opt := range opts { + opt(&cfg) + } + + return cfg +} + +// Instrument applies OpenTelemetry tracing and metrics to a go-redis +// UniversalClient (covering Redis and Valkey). The application owns the client; +// this helper only attaches hooks. +// +// The PII/cardinality guardrail is always enforced: db.statement (raw command, +// key, and values) is disabled on spans via WithDBStatement(false). +// +// Nil-safe: a nil client returns ErrNilClient and never panics. With no +// providers configured it attaches against the no-op providers, so telemetry +// being off never breaks the client. +func Instrument(client redis.UniversalClient, opts ...Option) error { + if client == nil { + return ErrNilClient + } + + cfg := newConfig(opts...) + + // Common attributes shared by tracing and metrics. db.system=redis is the + // value redisotel already uses; the shared list carries only bounded extras. + commonAttrs := make([]attribute.KeyValue, 0, 1+len(cfg.extraAttrs)) + commonAttrs = append(commonAttrs, cfg.extraAttrs...) + + tracingOpts := []redisotel.TracingOption{ + redisotel.WithTracerProvider(cfg.tracerProvider), + // GUARDRAIL (ADR-004, docs/metrics-contract.md): never attach the raw + // command / key / value (db.statement) to spans. redisotel enables it by + // default. + redisotel.WithDBStatement(false), + } + if len(commonAttrs) > 0 { + tracingOpts = append(tracingOpts, redisotel.WithAttributes(commonAttrs...)) + } + + if err := redisotel.InstrumentTracing(client, tracingOpts...); err != nil { + return err + } + + metricsOpts := []redisotel.MetricsOption{ + redisotel.WithMeterProvider(cfg.meterProvider), + } + if len(commonAttrs) > 0 { + metricsOpts = append(metricsOpts, redisotel.WithAttributes(commonAttrs...)) + } + + return redisotel.InstrumentMetrics(client, metricsOpts...) +} + +// System returns the db.system value redisotel emits for both Redis and Valkey. +// Exposed so callers building dashboards/tests can reference the canonical value +// without hardcoding it. +func System() string { + return constant.DBSystemRedis +} diff --git a/redisobs/redisobs_test.go b/redisobs/redisobs_test.go new file mode 100644 index 0000000..70c0b2a --- /dev/null +++ b/redisobs/redisobs_test.go @@ -0,0 +1,121 @@ +//go:build unit + +package redisobs + +import ( + "context" + "testing" + "time" + + "github.com/redis/go-redis/v9" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" +) + +// newRedisHarness builds real OTel SDK providers plus an in-memory span exporter +// so the test can assert on the spans redisotel produces without an external +// backend. +func newRedisHarness(t *testing.T) (*sdkmetric.MeterProvider, *sdktrace.TracerProvider, *tracetest.InMemoryExporter) { + t.Helper() + + mp := sdkmetric.NewMeterProvider() + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + spanExp := tracetest.NewInMemoryExporter() + tp := sdktrace.NewTracerProvider(sdktrace.WithSyncer(spanExp)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + + return mp, tp, spanExp +} + +// newUnreachableClient returns a go-redis UniversalClient pointed at an address +// that will fail to dial fast. redisotel's ProcessHook still creates the command +// span before the dial is attempted, so span attributes are observable without a +// live server. +func newUnreachableClient(t *testing.T) redis.UniversalClient { + t.Helper() + + c := redis.NewUniversalClient(&redis.UniversalOptions{ + Addrs: []string{"127.0.0.1:1"}, // nothing listens here + DialTimeout: 50 * time.Millisecond, + MaxRetries: -1, + }) + t.Cleanup(func() { _ = c.Close() }) + + return c +} + +// TestInstrument_AppliesHooksAndCreatesSpan verifies the helper wires tracing so +// a command produces a redis span, and does so without error. +func TestInstrument_AppliesHooksAndCreatesSpan(t *testing.T) { + mp, tp, spanExp := newRedisHarness(t) + + client := newUnreachableClient(t) + + err := Instrument(client, WithMeterProvider(mp), WithTracerProvider(tp)) + require.NoError(t, err) + + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + + // The command will fail to connect; we only care that a span was produced. + _ = client.Get(ctx, "some-key").Err() + + spans := spanExp.GetSpans() + require.NotEmpty(t, spans, "expected redisotel to create a command span") +} + +// TestInstrument_NeverEmitsCommandOrKeyAsAttribute is the PII/cardinality +// guardrail: redisotel attaches db.statement (the raw command incl. key/values) +// by default. The helper MUST disable it, so no span carries the command text, +// key, or value as an attribute (docs/metrics-contract.md FORBIDDEN list). +func TestInstrument_NeverEmitsCommandOrKeyAsAttribute(t *testing.T) { + _, tp, spanExp := newRedisHarness(t) + + client := newUnreachableClient(t) + + require.NoError(t, Instrument(client, WithTracerProvider(tp))) + + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + + const secretKey = "pix:key:123.456.789-00" + const secretVal = "super-secret-token" + + _ = client.Set(ctx, secretKey, secretVal, 0).Err() + _ = client.Get(ctx, secretKey).Err() + + spans := spanExp.GetSpans() + require.NotEmpty(t, spans) + + for _, s := range spans { + for _, kv := range s.Attributes { + key := string(kv.Key) + val := kv.Value.Emit() + + assert.NotEqual(t, "db.statement", key, + "db.statement present; WithDBStatement(false) guardrail failed") + assert.NotEqual(t, "db.query.text", key, "db.query.text present on redis span") + + assert.NotContains(t, val, secretKey, "span attribute leaked redis key: %s=%s", key, val) + assert.NotContains(t, val, secretVal, "span attribute leaked redis value: %s=%s", key, val) + assert.NotContains(t, val, "123.456.789-00", "span attribute leaked PII: %s=%s", key, val) + } + } +} + +// TestInstrument_NilClientReturnsError verifies the helper is nil-safe. +func TestInstrument_NilClientReturnsError(t *testing.T) { + err := Instrument(nil) + require.Error(t, err) +} + +// TestInstrument_NoTelemetryDoesNotError verifies that with no providers the +// helper still succeeds (degrading to no-op providers) and never breaks the app. +func TestInstrument_NoTelemetryDoesNotError(t *testing.T) { + client := newUnreachableClient(t) + require.NoError(t, Instrument(client)) +} diff --git a/sqlobs/doc.go b/sqlobs/doc.go new file mode 100644 index 0000000..2b5216f --- /dev/null +++ b/sqlobs/doc.go @@ -0,0 +1,39 @@ +// Package sqlobs provides thin, nil-safe helpers that add OpenTelemetry +// instrumentation to a database/sql handle the application already owns. It +// covers PostgreSQL (via pgx/stdlib or any database/sql driver) and +// MySQL/MariaDB — the two share the database/sql layer, so a single helper +// instruments both, differing only in the parameterizable db.system.name value. +// +// # Boundary (ADR-002, ADR-007) +// +// This package does NOT own connections: it never calls sql.Open on the +// caller's behalf as part of ownership, never builds a dbresolver, and never +// manages a pool's lifecycle. It wraps a *sql.DB (or a driver.Connector) the +// caller created and returns an instrumented handle. For a read/write split +// built on github.com/bxcodec/dbresolver, the caller MUST instrument EACH +// underlying *sql.DB (primary and every replica) with InstrumentDB BEFORE +// passing them to dbresolver.New — the resolver itself is not wrappable because +// it exposes no driver.Driver. +// +// # Emitted telemetry +// +// The wrapped handle emits the OpenTelemetry semantic-convention metric +// db.client.operation.duration (Float64 histogram, unit "s") via XSAM/otelsql, +// carrying db.system.name (postgresql | mysql), db.operation.name, and — when +// the driver/DSN supply them — db.collection.name / db.namespace, plus error.type +// on failures. An optional low-cardinality pool-role attribute (primary | +// replica) can be added with WithPoolRole (ADR-002). +// +// # PII / cardinality guardrail (docs/metrics-contract.md) +// +// otelsql captures db.query.text on spans by default. This package disables that +// unconditionally (SpanOptions.DisableQuery) and never enables SQLCommenter, so +// no query text, SQL statement, or bind parameter is ever emitted as a span or +// metric attribute. This is enforced by tests. +// +// # No-op degradation (ADR-008) +// +// With no MeterProvider/TracerProvider supplied the helper still returns a +// working *sql.DB; instrumentation degrades to the OTel no-op providers. The +// helper never panics and never breaks the caller's connection. +package sqlobs diff --git a/sqlobs/sqlobs.go b/sqlobs/sqlobs.go new file mode 100644 index 0000000..52463d0 --- /dev/null +++ b/sqlobs/sqlobs.go @@ -0,0 +1,258 @@ +package sqlobs + +import ( + "context" + "database/sql" + "database/sql/driver" + "errors" + + constant "github.com/LerianStudio/lib-observability/v2/constants" + "github.com/XSAM/otelsql" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + "go.opentelemetry.io/otel/trace" +) + +// System identifies the database management system for the db.system.name +// attribute. Only the values the platform actually runs are exposed, keeping the +// label bounded. +type System string + +const ( + // SystemPostgreSQL is the db.system.name value for PostgreSQL (incl. pgx/stdlib). + SystemPostgreSQL System = System(constant.DBSystemPostgreSQL) + // SystemMySQL is the db.system.name value for MySQL and MariaDB (same driver layer). + SystemMySQL System = "mysql" +) + +// PoolRole is the optional low-cardinality attribute distinguishing a primary +// (read/write) pool from a replica (read-only) pool in a read/write split +// (ADR-002). Bounded to two values. +type PoolRole string + +const ( + // PoolRolePrimary marks the read/write pool. + PoolRolePrimary PoolRole = "primary" + // PoolRoleReplica marks a read-only replica pool. + PoolRoleReplica PoolRole = "replica" +) + +// poolRoleAttrKey is the metric/span attribute key carrying the PoolRole. It is +// namespaced under db.sql to signal it is a library extension, not a semconv +// attribute. +const poolRoleAttrKey = "db.sql.pool.role" + +// ErrNilDB is returned by InstrumentDB / RegisterDBStatsMetrics when the caller +// passes a nil *sql.DB. +var ErrNilDB = errors.New("sqlobs: nil *sql.DB") + +// config holds resolved helper options. +type config struct { + meterProvider metric.MeterProvider + tracerProvider trace.TracerProvider + poolRole PoolRole + dsn string + extraAttrs []attribute.KeyValue +} + +// Option configures the SQL instrumentation helpers. +type Option func(*config) + +// WithMeterProvider sets the MeterProvider used for the duration/pool metrics. +// When unset, the global (no-op unless SetMeterProvider was called) provider is +// used, so metrics degrade to no-op rather than breaking the caller. +func WithMeterProvider(mp metric.MeterProvider) Option { + return func(c *config) { + if mp != nil { + c.meterProvider = mp + } + } +} + +// WithTracerProvider sets the TracerProvider used for query spans. When unset, +// the global provider is used. +func WithTracerProvider(tp trace.TracerProvider) Option { + return func(c *config) { + if tp != nil { + c.tracerProvider = tp + } + } +} + +// WithPoolRole adds the optional primary/replica attribute (ADR-002) to every +// emitted metric and span, giving read-vs-write visibility for a dbresolver +// split. Empty values are ignored. +func WithPoolRole(role PoolRole) Option { + return func(c *config) { + if role != "" { + c.poolRole = role + } + } +} + +// WithDSN supplies the data source name used to re-open the connection through +// the instrumented driver when instrumenting an existing *sql.DB via +// InstrumentDB. A *sql.DB does not expose its DSN, so it must be provided here; +// when omitted, the driver is re-opened with an empty DSN (valid for drivers +// that resolve configuration elsewhere). Prefer Open when the DSN is known at +// construction time. +func WithDSN(dsn string) Option { + return func(c *config) { + c.dsn = dsn + } +} + +// WithAttributes appends additional low-cardinality, PII-free attributes to +// every metric and span. Callers are responsible for keeping these bounded; +// query text / parameters / IDs are FORBIDDEN (docs/metrics-contract.md). +func WithAttributes(attrs ...attribute.KeyValue) Option { + return func(c *config) { + c.extraAttrs = append(c.extraAttrs, attrs...) + } +} + +func newConfig(opts ...Option) config { + cfg := config{ + meterProvider: otel.GetMeterProvider(), + tracerProvider: otel.GetTracerProvider(), + } + + for _, opt := range opts { + opt(&cfg) + } + + return cfg +} + +// baseAttributes builds the attribute slice applied to every span and metric: +// db.system.name plus the optional pool role and caller extras. Query text is +// never included. +func (c config) baseAttributes(system System) []attribute.KeyValue { + attrs := make([]attribute.KeyValue, 0, 2+len(c.extraAttrs)) + attrs = append(attrs, attribute.String("db.system.name", string(system))) + + if c.poolRole != "" { + attrs = append(attrs, attribute.String(poolRoleAttrKey, string(c.poolRole))) + } + + attrs = append(attrs, c.extraAttrs...) + + return attrs +} + +// otelsqlOptions translates the resolved config into otelsql options, always +// applying the PII/cardinality guardrail: db.query.text is suppressed on spans +// and SQLCommenter is never enabled, so no query text or bind parameters reach +// spans or metrics. +func (c config) otelsqlOptions(system System) []otelsql.Option { + return []otelsql.Option{ + otelsql.WithMeterProvider(c.meterProvider), + otelsql.WithTracerProvider(c.tracerProvider), + otelsql.WithAttributes(c.baseAttributes(system)...), + otelsql.WithSpanOptions(otelsql.SpanOptions{ + // GUARDRAIL (ADR-002 §3, docs/metrics-contract.md): never attach + // db.query.text to spans. otelsql captures it by default. + DisableQuery: true, + // Ping / RowsNext spans add cardinality/noise with no operational + // value for a duration signal; leave them off (also the otelsql + // default, set explicitly for intent). + Ping: false, + RowsNext: false, + }), + } +} + +// dsnConnector adapts a driver.Driver + DSN into a driver.Connector so an +// instrumented driver can back a fresh *sql.DB. It mirrors the stdlib's +// unexported dsnConnector. +type dsnConnector struct { + dsn string + driver driver.Driver +} + +func (c dsnConnector) Connect(context.Context) (driver.Conn, error) { + return c.driver.Open(c.dsn) +} + +func (c dsnConnector) Driver() driver.Driver { return c.driver } + +// InstrumentDB returns a new *sql.DB that wraps the same underlying driver as +// db with OpenTelemetry instrumentation, emitting db.client.operation.duration +// (seconds) tagged with db.system.name=system. +// +// IMPORTANT — separate connection pool: the returned *sql.DB is backed by a +// FRESH, independent connection pool (built via sql.OpenDB); it does NOT share +// the pool of the input db. To avoid two live pools against the same database, +// the caller MUST: +// 1. use ONLY the returned handle going forward, and Close() the original db; +// 2. re-apply any pool tuning (SetMaxOpenConns / SetMaxIdleConns / +// SetConnMaxLifetime / SetConnMaxIdleTime) on the RETURNED handle — those +// settings are per-*sql.DB and are NOT carried over from the original. +// The caller keeps ownership of the connection lifecycle; this helper only adds +// instrumentation. +// +// Because a *sql.DB does not expose its DSN, supply it via WithDSN when the +// driver needs it to open connections (most do). When no DSN is given the +// underlying driver is re-opened with an empty DSN. +// +// For a dbresolver read/write split, call InstrumentDB on EACH *sql.DB (primary +// and every replica) BEFORE building the resolver (ADR-002); the resolver is not +// wrappable. +// +// Nil-safe: a nil db returns ErrNilDB and never panics. With no telemetry +// providers configured the returned handle is still a working *sql.DB (no-op +// instrumentation), so telemetry being off never breaks the caller. +func InstrumentDB(db *sql.DB, system System, opts ...Option) (*sql.DB, error) { + if db == nil { + return nil, ErrNilDB + } + + cfg := newConfig(opts...) + + wrappedDriver := otelsql.WrapDriver(db.Driver(), cfg.otelsqlOptions(system)...) + + // Prefer the context-aware connector API when the underlying (and thus the + // wrapped) driver implements DriverContext, so those features are preserved; + // otherwise fall back to a plain DSN connector. + if dc, ok := wrappedDriver.(driver.DriverContext); ok { + connector, err := dc.OpenConnector(cfg.dsn) + if err != nil { + return nil, err + } + + return sql.OpenDB(connector), nil + } + + return sql.OpenDB(dsnConnector{dsn: cfg.dsn, driver: wrappedDriver}), nil +} + +// Open opens a new instrumented *sql.DB directly from a driver name and DSN, +// applying the same guardrails as InstrumentDB. Prefer this when the DSN is +// known at construction time; it avoids opening an uninstrumented handle first. +// The caller still owns the returned handle's lifecycle. +func Open(driverName, dsn string, system System, opts ...Option) (*sql.DB, error) { + cfg := newConfig(opts...) + + return otelsql.Open(driverName, dsn, cfg.otelsqlOptions(system)...) +} + +// RegisterDBStatsMetrics registers the opt-in, low-cardinality connection-pool +// metrics (db.sql.connection.*: open/idle/max_open/wait/...) for an instrumented +// *sql.DB, tagged with db.system.name=system and any configured pool role. These +// are safe to always enable — the pool exposes a fixed, tiny set of gauges. Call +// Unregister on the returned Registration when the pool is closed. +// +// Nil-safe: a nil db returns ErrNilDB. +func RegisterDBStatsMetrics(db *sql.DB, system System, opts ...Option) (metric.Registration, error) { + if db == nil { + return nil, ErrNilDB + } + + cfg := newConfig(opts...) + + return otelsql.RegisterDBStatsMetrics(db, + otelsql.WithMeterProvider(cfg.meterProvider), + otelsql.WithAttributes(cfg.baseAttributes(system)...), + ) +} diff --git a/sqlobs/sqlobs_test.go b/sqlobs/sqlobs_test.go new file mode 100644 index 0000000..f5e5d46 --- /dev/null +++ b/sqlobs/sqlobs_test.go @@ -0,0 +1,354 @@ +//go:build unit + +package sqlobs + +import ( + "context" + "database/sql" + "database/sql/driver" + "io" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" +) + +// dbClientOperationDurationMetric is the semconv metric name emitted by otelsql +// for database client operation duration. Asserted here so the test fails loudly +// if the underlying instrument name ever drifts from the contract. +const dbClientOperationDurationMetric = "db.client.operation.duration" + +// --- Minimal in-memory database/sql driver for the harness -------------------- +// +// The lib does NOT own connections; these fakes exist only so the test can drive +// a real *sql.DB through otelsql and observe the emitted telemetry. They record +// nothing about the query text, mirroring how a production driver would be +// wrapped. + +type fakeDriver struct{} + +func (fakeDriver) Open(string) (driver.Conn, error) { return &fakeConn{}, nil } + +type fakeConn struct{} + +func (*fakeConn) Prepare(query string) (driver.Stmt, error) { return &fakeStmt{query: query}, nil } +func (*fakeConn) Close() error { return nil } +func (*fakeConn) Begin() (driver.Tx, error) { return &fakeTx{}, nil } + +// QueryContext lets the driver satisfy driver.QueryerContext so ExecContext / +// QueryContext flow straight through without the prepared-statement fallback. +func (*fakeConn) QueryContext(_ context.Context, _ string, _ []driver.NamedValue) (driver.Rows, error) { + return &fakeRows{}, nil +} + +func (*fakeConn) ExecContext(_ context.Context, _ string, _ []driver.NamedValue) (driver.Result, error) { + return driver.RowsAffected(0), nil +} + +func (*fakeConn) Ping(context.Context) error { return nil } + +type fakeStmt struct{ query string } + +func (*fakeStmt) Close() error { return nil } +func (*fakeStmt) NumInput() int { return 0 } +func (*fakeStmt) Exec([]driver.Value) (driver.Result, error) { return driver.RowsAffected(0), nil } +func (*fakeStmt) Query([]driver.Value) (driver.Rows, error) { return &fakeRows{}, nil } + +type fakeRows struct{} + +func (*fakeRows) Columns() []string { return []string{} } +func (*fakeRows) Close() error { return nil } +func (*fakeRows) Next([]driver.Value) error { return io.EOF } + +type fakeTx struct{} + +func (*fakeTx) Commit() error { return nil } +func (*fakeTx) Rollback() error { return nil } + +func init() { + sql.Register("sqlobs-fake", fakeDriver{}) +} + +// --- Harness ------------------------------------------------------------------ + +func newHarness(t *testing.T) (*sdkmetric.MeterProvider, *sdkmetric.ManualReader, *sdktrace.TracerProvider, *tracetest.InMemoryExporter) { + t.Helper() + + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + spanExp := tracetest.NewInMemoryExporter() + tp := sdktrace.NewTracerProvider(sdktrace.WithSyncer(spanExp)) + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + + return mp, reader, tp, spanExp +} + +// openFake opens a raw *sql.DB against the in-test fake driver, standing in for +// the connection the app already created before handing it to the helper. +func openFake(t *testing.T) *sql.DB { + t.Helper() + + db, err := sql.Open("sqlobs-fake", "fake://db") + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + + return db +} + +func collectDurationDataPoints(t *testing.T, reader *sdkmetric.ManualReader) []metricdata.HistogramDataPoint[float64] { + t.Helper() + + rm := &metricdata.ResourceMetrics{} + require.NoError(t, reader.Collect(context.Background(), rm)) + + var points []metricdata.HistogramDataPoint[float64] + + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name != dbClientOperationDurationMetric { + continue + } + + require.Equal(t, "s", m.Unit, "db duration must be seconds") + + h, ok := m.Data.(metricdata.Histogram[float64]) + require.True(t, ok, "expected float64 histogram, got %T", m.Data) + points = append(points, h.DataPoints...) + } + } + + return points +} + +func attrString(set attribute.Set, key string) (string, bool) { + v, ok := set.Value(attribute.Key(key)) + if !ok { + return "", false + } + + return v.AsString(), true +} + +// TestInstrumentDB_EmitsDurationForPostgres verifies a real query through the +// wrapped *sql.DB emits db.client.operation.duration in seconds with the +// parameterizable db.system.name and a db.operation.name attribute. +func TestInstrumentDB_EmitsDurationForPostgres(t *testing.T) { + mp, reader, tp, _ := newHarness(t) + + raw := openFake(t) + + db, err := InstrumentDB(raw, SystemPostgreSQL, + WithMeterProvider(mp), + WithTracerProvider(tp), + ) + require.NoError(t, err) + require.NotNil(t, db) + + _, err = db.ExecContext(context.Background(), "INSERT INTO accounts VALUES (1)") + require.NoError(t, err) + + points := collectDurationDataPoints(t, reader) + require.NotEmpty(t, points, "expected db.client.operation.duration to be recorded") + + var found bool + + for _, dp := range points { + system, ok := attrString(dp.Attributes, "db.system.name") + if !ok { + continue + } + + assert.Equal(t, "postgresql", system) + + _, hasOp := attrString(dp.Attributes, "db.operation.name") + assert.True(t, hasOp, "db.operation.name must be present") + + found = true + } + + assert.True(t, found, "at least one data point must carry db.system.name=postgresql") +} + +// TestInstrumentDB_SystemNameParameterizable verifies mysql is emitted when the +// caller selects the MySQL system, proving the same helper covers MySQL/MariaDB. +func TestInstrumentDB_SystemNameParameterizable(t *testing.T) { + mp, reader, tp, _ := newHarness(t) + + db, err := InstrumentDB(openFake(t), SystemMySQL, + WithMeterProvider(mp), + WithTracerProvider(tp), + ) + require.NoError(t, err) + + _, err = db.ExecContext(context.Background(), "SELECT 1") + require.NoError(t, err) + + points := collectDurationDataPoints(t, reader) + require.NotEmpty(t, points) + + var systems []string + for _, dp := range points { + if s, ok := attrString(dp.Attributes, "db.system.name"); ok { + systems = append(systems, s) + } + } + + assert.Contains(t, systems, "mysql") +} + +// TestInstrumentDB_NeverEmitsQueryText is the PII/cardinality guardrail: no +// metric data point and no span may carry query text, SQL statement, or bind +// parameters as an attribute (docs/metrics-contract.md FORBIDDEN list). +func TestInstrumentDB_NeverEmitsQueryText(t *testing.T) { + mp, reader, tp, spanExp := newHarness(t) + + db, err := InstrumentDB(openFake(t), SystemPostgreSQL, + WithMeterProvider(mp), + WithTracerProvider(tp), + ) + require.NoError(t, err) + + const secret = "SELECT secret_column FROM pix_keys WHERE cpf = '123.456.789-00'" + + rows, err := db.QueryContext(context.Background(), secret) + require.NoError(t, err) + require.NoError(t, rows.Close()) + + forbidden := []string{ + "db.statement", + "db.query.text", + "db.query.parameter", + "sql", + "statement", + } + + // Metric side. + points := collectDurationDataPoints(t, reader) + require.NotEmpty(t, points) + + for _, dp := range points { + for _, kv := range dp.Attributes.ToSlice() { + key := string(kv.Key) + val := kv.Value.Emit() + + for _, bad := range forbidden { + assert.NotEqual(t, bad, key, "forbidden label %q present on db duration metric", bad) + } + + assert.NotContains(t, val, "secret_column", "metric label leaked query text: %s=%s", key, val) + assert.NotContains(t, val, "cpf", "metric label leaked PII column: %s=%s", key, val) + } + } + + // Span side (otelsql captures db.query.text on spans by default; the helper + // MUST disable that). + for _, s := range spanExp.GetSpans() { + for _, kv := range s.Attributes { + key := string(kv.Key) + val := kv.Value.Emit() + + assert.NotEqual(t, "db.query.text", key, "span carries db.query.text; DisableQuery guardrail failed") + assert.NotEqual(t, "db.statement", key, "span carries db.statement") + assert.NotContains(t, val, "secret_column", "span attribute leaked query text: %s=%s", key, val) + } + } +} + +// TestInstrumentDB_PoolRoleAttribute verifies the optional primary/replica +// distinguishing attribute (ADR-002) is emitted on the metric when supplied, +// bounded to a low-cardinality value. +func TestInstrumentDB_PoolRoleAttribute(t *testing.T) { + mp, reader, tp, _ := newHarness(t) + + db, err := InstrumentDB(openFake(t), SystemPostgreSQL, + WithMeterProvider(mp), + WithTracerProvider(tp), + WithPoolRole(PoolRoleReplica), + ) + require.NoError(t, err) + + _, err = db.ExecContext(context.Background(), "SELECT 1") + require.NoError(t, err) + + points := collectDurationDataPoints(t, reader) + require.NotEmpty(t, points) + + var found bool + for _, dp := range points { + if role, ok := attrString(dp.Attributes, poolRoleAttrKey); ok { + assert.Equal(t, "replica", role) + found = true + } + } + + assert.True(t, found, "pool role attribute must be present when WithPoolRole is set") +} + +// TestInstrumentDB_NilDBReturnsError verifies the helper is nil-safe: a nil +// input never panics and never returns a usable-but-broken handle. +func TestInstrumentDB_NilDBReturnsError(t *testing.T) { + db, err := InstrumentDB(nil, SystemPostgreSQL) + require.Error(t, err) + assert.Nil(t, db) +} + +// TestInstrumentDB_NoTelemetryReturnsOriginal verifies that with no providers +// configured the helper degrades to a no-op passthrough, returning a working +// *sql.DB (the original) so the app is never broken by telemetry being off. +func TestInstrumentDB_NoTelemetryReturnsOriginal(t *testing.T) { + raw := openFake(t) + + db, err := InstrumentDB(raw, SystemPostgreSQL) + require.NoError(t, err) + require.NotNil(t, db) + + // The returned handle must still be usable. + _, err = db.ExecContext(context.Background(), "SELECT 1") + require.NoError(t, err) +} + +// TestRegisterDBStatsMetrics_EmitsPoolMetrics verifies the opt-in pool metrics +// helper registers low-cardinality connection-pool gauges. +func TestRegisterDBStatsMetrics_EmitsPoolMetrics(t *testing.T) { + mp, reader, tp, _ := newHarness(t) + + db, err := InstrumentDB(openFake(t), SystemPostgreSQL, + WithMeterProvider(mp), + WithTracerProvider(tp), + ) + require.NoError(t, err) + + reg, err := RegisterDBStatsMetrics(db, SystemPostgreSQL, WithMeterProvider(mp)) + require.NoError(t, err) + t.Cleanup(func() { _ = reg.Unregister() }) + + // Force at least one connection so the pool has observable state. + _, err = db.ExecContext(context.Background(), "SELECT 1") + require.NoError(t, err) + + rm := &metricdata.ResourceMetrics{} + require.NoError(t, reader.Collect(context.Background(), rm)) + + var names []string + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + names = append(names, m.Name) + } + } + + var hasPoolMetric bool + for _, n := range names { + if n == "db.sql.connection.open" || n == "db.sql.connection.max_open" { + hasPoolMetric = true + } + } + + assert.True(t, hasPoolMetric, "expected db.sql.connection.* pool metrics, got: %v", names) +} diff --git a/telemetrycore/metrics_collector.go b/telemetrycore/metrics_collector.go new file mode 100644 index 0000000..3c31705 --- /dev/null +++ b/telemetrycore/metrics_collector.go @@ -0,0 +1,143 @@ +// Package telemetrycore holds the transport-agnostic, Fiber-free telemetry +// primitives shared by the HTTP (middleware) and gRPC (grpcmiddleware) packages. +// +// It exists so that the background system-metrics collector is a single +// process-wide singleton regardless of whether an application wires up the HTTP +// middleware, the gRPC interceptors, or both. Keeping this logic out of the +// Fiber-importing middleware package also lets Fiber-v2 applications consume the +// gRPC interceptors and the collector without pulling in Fiber v3. +package telemetrycore + +import ( + "context" + "errors" + "os" + "sync" + "time" + + observability "github.com/LerianStudio/lib-observability/v2" + "github.com/LerianStudio/lib-observability/v2/runtime" + "github.com/LerianStudio/lib-observability/v2/tracing" +) + +// DefaultMetricsCollectionInterval is the default interval for collecting system metrics. +// Can be overridden via METRICS_COLLECTION_INTERVAL environment variable. +const DefaultMetricsCollectionInterval = 5 * time.Second + +// Metrics collector singleton state. +var ( + metricsCollectorOnce = &sync.Once{} + metricsCollectorShutdown chan struct{} + metricsCollectorMu sync.Mutex + metricsCollectorStarted bool + metricsCollectorInitErr error +) + +// telemetryRuntimeLogger returns the runtime logger from the telemetry, or nil. +func telemetryRuntimeLogger(tl *tracing.Telemetry) runtime.Logger { + if tl == nil { + return nil + } + + return tl.Logger +} + +// EnsureMetricsCollector lazily starts the background metrics collector singleton +// for the given telemetry. It is safe to call from both the HTTP middleware and +// the gRPC interceptors: only the first successful call starts the collector, so +// an application wiring up both transports still runs exactly one collector. +func EnsureMetricsCollector(tl *tracing.Telemetry) error { + if tl == nil { + return nil + } + + if tl.MeterProvider == nil { + return nil + } + + metricsCollectorMu.Lock() + defer metricsCollectorMu.Unlock() + + if metricsCollectorStarted { + return nil + } + + if metricsCollectorInitErr != nil { + metricsCollectorOnce = &sync.Once{} + metricsCollectorInitErr = nil + } + + metricsCollectorOnce.Do(func() { + factory := tl.MetricsFactory + if factory == nil { + metricsCollectorInitErr = errors.New("telemetry MetricsFactory is nil, cannot start system metrics collector") + return + } + + shutdown := make(chan struct{}) + metricsCollectorShutdown = shutdown + ticker := time.NewTicker(getMetricsCollectionInterval()) + + runtime.SafeGoWithContextAndComponent( + context.Background(), + telemetryRuntimeLogger(tl), + "http", + "metrics_collector", + runtime.KeepRunning, + func(_ context.Context) { + observability.GetCPUUsage(context.Background(), factory) + observability.GetMemUsage(context.Background(), factory) + + for { + select { + case <-shutdown: + ticker.Stop() + return + case <-ticker.C: + observability.GetCPUUsage(context.Background(), factory) + observability.GetMemUsage(context.Background(), factory) + } + } + }, + ) + + metricsCollectorStarted = true + }) + + return metricsCollectorInitErr +} + +// getMetricsCollectionInterval returns the metrics collection interval. +// Can be configured via METRICS_COLLECTION_INTERVAL environment variable. +// Accepts Go duration format (e.g., "10s", "1m", "500ms"). +// Falls back to DefaultMetricsCollectionInterval if not set or invalid. +func getMetricsCollectionInterval() time.Duration { + if envInterval := os.Getenv("METRICS_COLLECTION_INTERVAL"); envInterval != "" { + if parsed, err := time.ParseDuration(envInterval); err == nil && parsed > 0 { + return parsed + } + } + + return DefaultMetricsCollectionInterval +} + +// StopMetricsCollector stops the background metrics collector goroutine. +// Should be called during application shutdown for graceful cleanup. +// After calling this function, the collector can be restarted by new requests. +// +// Implementation note: This function intentionally resets sync.Once to a new instance +// to allow the collector to be restarted after being stopped. This is an unusual but +// intentional pattern - the mutex ensures thread-safety during the reset operation, +// preventing race conditions between Stop and subsequent Start calls. +func StopMetricsCollector() { + metricsCollectorMu.Lock() + defer metricsCollectorMu.Unlock() + + if metricsCollectorStarted && metricsCollectorShutdown != nil { + close(metricsCollectorShutdown) + + metricsCollectorStarted = false + metricsCollectorOnce = &sync.Once{} + metricsCollectorInitErr = nil + } +} diff --git a/telemetrycore/metrics_collector_test.go b/telemetrycore/metrics_collector_test.go new file mode 100644 index 0000000..0cbb0f7 --- /dev/null +++ b/telemetrycore/metrics_collector_test.go @@ -0,0 +1,132 @@ +//go:build unit + +package telemetrycore + +import ( + "sync" + "testing" + "time" + + "github.com/LerianStudio/lib-observability/v2/metrics" + "github.com/LerianStudio/lib-observability/v2/tracing" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" +) + +// TestGetMetricsCollectionInterval tests the getMetricsCollectionInterval function. +func TestGetMetricsCollectionInterval(t *testing.T) { + tests := []struct { + name string + envValue string + expected time.Duration + }{ + { + name: "default when not set", + envValue: "", + expected: DefaultMetricsCollectionInterval, + }, + { + name: "valid duration in seconds", + envValue: "10s", + expected: 10 * time.Second, + }, + { + name: "valid duration in milliseconds", + envValue: "500ms", + expected: 500 * time.Millisecond, + }, + { + name: "valid duration in minutes", + envValue: "1m", + expected: 1 * time.Minute, + }, + { + name: "invalid format falls back to default", + envValue: "invalid", + expected: DefaultMetricsCollectionInterval, + }, + { + name: "zero value falls back to default", + envValue: "0s", + expected: DefaultMetricsCollectionInterval, + }, + { + name: "negative value falls back to default", + envValue: "-5s", + expected: DefaultMetricsCollectionInterval, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if tt.envValue != "" { + t.Setenv("METRICS_COLLECTION_INTERVAL", tt.envValue) + } else { + t.Setenv("METRICS_COLLECTION_INTERVAL", "") + } + + result := getMetricsCollectionInterval() + assert.Equal(t, tt.expected, result) + }) + } +} + +func resetMetricsCollectorState() { + metricsCollectorMu.Lock() + defer metricsCollectorMu.Unlock() + + if metricsCollectorStarted && metricsCollectorShutdown != nil { + close(metricsCollectorShutdown) + time.Sleep(50 * time.Millisecond) + } + + metricsCollectorShutdown = nil + metricsCollectorStarted = false + metricsCollectorOnce = &sync.Once{} + metricsCollectorInitErr = nil +} + +func TestEnsureMetricsCollector_ReturnsErrorWhenMetricsFactoryNil(t *testing.T) { + resetMetricsCollectorState() + t.Cleanup(resetMetricsCollectorState) + + tl := &tracing.Telemetry{ + TelemetryConfig: tracing.TelemetryConfig{LibraryName: "test-library", EnableTelemetry: true}, + MeterProvider: sdkmetric.NewMeterProvider(), + } + + err := EnsureMetricsCollector(tl) + require.Error(t, err) + assert.Contains(t, err.Error(), "MetricsFactory is nil") + assert.False(t, metricsCollectorStarted) +} + +func TestEnsureMetricsCollector_NoMeterProviderReturnsNil(t *testing.T) { + resetMetricsCollectorState() + t.Cleanup(resetMetricsCollectorState) + + tl := &tracing.Telemetry{} + require.NoError(t, EnsureMetricsCollector(tl)) + assert.False(t, metricsCollectorStarted) +} + +func TestStopMetricsCollector_AllowsRestart(t *testing.T) { + resetMetricsCollectorState() + t.Cleanup(resetMetricsCollectorState) + + tl := &tracing.Telemetry{ + TelemetryConfig: tracing.TelemetryConfig{LibraryName: "test-library", EnableTelemetry: true}, + MeterProvider: sdkmetric.NewMeterProvider(), + MetricsFactory: metrics.NewNopFactory(), + } + + require.NoError(t, EnsureMetricsCollector(tl)) + assert.True(t, metricsCollectorStarted) + + StopMetricsCollector() + assert.False(t, metricsCollectorStarted) + + require.NoError(t, EnsureMetricsCollector(tl)) + assert.True(t, metricsCollectorStarted) +} diff --git a/tracing/otel.go b/tracing/otel.go index 94c42c5..3436548 100644 --- a/tracing/otel.go +++ b/tracing/otel.go @@ -12,15 +12,14 @@ import ( "reflect" "strconv" "strings" + "time" "unicode/utf8" - observability "github.com/LerianStudio/lib-observability/v2" "github.com/LerianStudio/lib-observability/v2/assert" constant "github.com/LerianStudio/lib-observability/v2/constants" "github.com/LerianStudio/lib-observability/v2/log" "github.com/LerianStudio/lib-observability/v2/metrics" - "github.com/LerianStudio/lib-observability/v2/redaction" - "github.com/gofiber/fiber/v3" + otelruntime "go.opentelemetry.io/contrib/instrumentation/runtime" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/codes" @@ -69,9 +68,20 @@ type TelemetryConfig struct { CollectorExporterEndpoint string EnableTelemetry bool InsecureExporter bool - Logger log.Logger - Propagator propagation.TextMapPropagator - Redactor *Redactor + // EnableRuntimeMetrics turns on the Go runtime instrumentation + // (go.opentelemetry.io/contrib/instrumentation/runtime), emitting go.* + // runtime metrics (heap, GC, goroutines) through this instance's + // MeterProvider. It follows the Go zero-value convention: default false, + // opt-in true. It is honored only when EnableTelemetry is also true and a + // real MeterProvider exists; with telemetry disabled or a noop provider it + // degrades to a no-op. Applications that want runtime metrics on by default + // should set this to true explicitly at their bootstrap - keeping the + // zero-value off avoids surprising callers who construct TelemetryConfig + // partially and inheriting a background collector they did not request. + EnableRuntimeMetrics bool + Logger log.Logger + Propagator propagation.TextMapPropagator + Redactor *Redactor } // Telemetry holds configured OpenTelemetry providers and lifecycle handlers. @@ -250,6 +260,12 @@ func initExporters(ctx context.Context, cfg TelemetryConfig) (*Telemetry, error) return nil, err } + // Start Go runtime instrumentation on the real MeterProvider when opted in. + // Best-effort: a failure here is logged inside the helper and never aborts + // telemetry bring-up, since runtime metrics are auxiliary to request-path + // observability. + startRuntimeMetrics(cfg, mp) + shutdown, shutdownCtx := buildShutdownHandlers(cfg.Logger, mp, tp, lp, tExp, mExp, lExp) return &Telemetry{ @@ -263,6 +279,51 @@ func initExporters(ctx context.Context, cfg TelemetryConfig) (*Telemetry, error) }, nil } +// runtimeMinReadMemStatsInterval is the minimum interval between the relatively +// expensive runtime.ReadMemStats() calls made by the Go runtime instrumentation. +const runtimeMinReadMemStatsInterval = 15 * time.Second + +// startRuntimeMetrics registers the Go runtime instrumentation +// (go.opentelemetry.io/contrib/instrumentation/runtime) against the supplied +// MeterProvider when cfg.EnableRuntimeMetrics is set. It returns true when the +// instrumentation was started, false when it was skipped (toggle off, nil +// MeterProvider) or failed to register. +// +// It is best-effort and never panics: a nil MeterProvider or a Start error is +// logged (when a logger is available) and reported via the false return, so +// telemetry bring-up proceeds regardless. The contrib instrumentation registers +// asynchronous callbacks on the MeterProvider's meter; there is no separate +// goroutine to shut down, so teardown follows the MeterProvider's own shutdown. +func startRuntimeMetrics(cfg TelemetryConfig, mp *sdkmetric.MeterProvider) bool { + if !cfg.EnableRuntimeMetrics { + return false + } + + if mp == nil { + if cfg.Logger != nil { + cfg.Logger.Log(context.Background(), log.LevelWarn, + "runtime metrics requested but MeterProvider is nil; skipping") + } + + return false + } + + err := otelruntime.Start( + otelruntime.WithMeterProvider(mp), + otelruntime.WithMinimumReadMemStatsInterval(runtimeMinReadMemStatsInterval), + ) + if err != nil { + if cfg.Logger != nil { + cfg.Logger.Log(context.Background(), log.LevelError, + "failed to start Go runtime metrics", log.Err(err)) + } + + return false + } + + return true +} + // newNoopTelemetry creates a Telemetry instance with no-op providers (no exporters). // This is used when telemetry is disabled or when the collector endpoint is empty, // ensuring global OTEL providers are safe no-ops that do not leak goroutines. @@ -736,27 +797,6 @@ func truncateUTF8(s string, maxBytes int) string { return s } -// SetSpanAttributeForParam adds a request parameter attribute to the current context bag. -// Sensitive parameter names (as determined by redaction.IsSensitiveField) are masked. -func SetSpanAttributeForParam(c fiber.Ctx, param, value, entityName string) { - if c == nil { - return - } - - spanAttrKey := "app.request." + param - if entityName != "" && param == "id" { - spanAttrKey = "app.request." + entityName + "_id" - } - - // Mask value if the parameter name is considered sensitive - attrValue := value - if redaction.IsSensitiveField(param) { - attrValue = "[REDACTED]" - } - - c.SetContext(observability.ContextWithSpanAttributes(c.Context(), attribute.String(spanAttrKey, attrValue))) -} - // InjectTraceContext injects trace context into a generic text map carrier. func InjectTraceContext(ctx context.Context, carrier propagation.TextMapCarrier) { if carrier == nil { @@ -784,20 +824,6 @@ func InjectHTTPContext(ctx context.Context, headers http.Header) { InjectTraceContext(ctx, propagation.HeaderCarrier(headers)) } -// ExtractHTTPContext extracts trace headers from a Fiber request. -func ExtractHTTPContext(ctx context.Context, c fiber.Ctx) context.Context { - if c == nil { - return ctx - } - - carrier := propagation.HeaderCarrier{} - for key, value := range c.Request().Header.All() { - carrier.Set(string(key), string(value)) - } - - return ExtractTraceContext(ctx, carrier) -} - // InjectGRPCContext injects trace context into gRPC metadata. func InjectGRPCContext(ctx context.Context, md metadata.MD) metadata.MD { if md == nil { diff --git a/tracing/runtime_metrics_test.go b/tracing/runtime_metrics_test.go new file mode 100644 index 0000000..ba18dca --- /dev/null +++ b/tracing/runtime_metrics_test.go @@ -0,0 +1,121 @@ +//go:build unit + +package tracing + +import ( + "context" + "strings" + "testing" + + "github.com/LerianStudio/lib-observability/v2/log" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" +) + +// collectMetricNames gathers every metric name emitted by a ManualReader-backed +// MeterProvider, used to assert the runtime instrumentation registered its +// go.* instruments. +func collectMetricNames(t *testing.T, reader *sdkmetric.ManualReader) []string { + t.Helper() + + rm := &metricdata.ResourceMetrics{} + require.NoError(t, reader.Collect(context.Background(), rm)) + + var names []string + + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + names = append(names, m.Name) + } + } + + return names +} + +func hasRuntimeMetric(names []string) bool { + for _, n := range names { + // contrib/runtime emits instruments under the "go." namespace + // (e.g. go.memory.used, go.goroutine.count) plus process.runtime.*. + if strings.HasPrefix(n, "go.") || strings.HasPrefix(n, "process.runtime.") { + return true + } + } + + return false +} + +// TestStartRuntimeMetrics_Disabled verifies the helper is a no-op when the +// EnableRuntimeMetrics toggle is off: no go.* instruments are registered. +func TestStartRuntimeMetrics_Disabled(t *testing.T) { + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + cfg := TelemetryConfig{ + LibraryName: "test-lib", + EnableTelemetry: true, + EnableRuntimeMetrics: false, + Logger: log.NewNop(), + } + + started := startRuntimeMetrics(cfg, mp) + assert.False(t, started, "runtime metrics must not start when disabled") + + names := collectMetricNames(t, reader) + assert.False(t, hasRuntimeMetric(names), + "no go.*/process.runtime.* metrics expected when disabled: %v", names) +} + +// TestStartRuntimeMetrics_Enabled verifies the helper registers the contrib +// runtime instruments when EnableRuntimeMetrics is on, so go.* metrics appear. +func TestStartRuntimeMetrics_Enabled(t *testing.T) { + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + t.Cleanup(func() { _ = mp.Shutdown(context.Background()) }) + + cfg := TelemetryConfig{ + LibraryName: "test-lib", + EnableTelemetry: true, + EnableRuntimeMetrics: true, + Logger: log.NewNop(), + } + + started := startRuntimeMetrics(cfg, mp) + require.True(t, started, "runtime metrics must start when enabled") + + names := collectMetricNames(t, reader) + assert.True(t, hasRuntimeMetric(names), + "expected go.*/process.runtime.* metrics after starting runtime instrumentation: %v", names) +} + +// TestStartRuntimeMetrics_NilMeterProviderIsSafe verifies the helper degrades +// to a no-op (never panics) when the MeterProvider is nil. +func TestStartRuntimeMetrics_NilMeterProviderIsSafe(t *testing.T) { + cfg := TelemetryConfig{ + LibraryName: "test-lib", + EnableTelemetry: true, + EnableRuntimeMetrics: true, + Logger: log.NewNop(), + } + + assert.NotPanics(t, func() { + started := startRuntimeMetrics(cfg, nil) + assert.False(t, started, "nil MeterProvider must not start runtime metrics") + }) +} + +// TestNewTelemetry_RuntimeMetricsConfigFieldDefaultsOff documents the zero-value +// Go convention: EnableRuntimeMetrics defaults to false unless explicitly set. +func TestNewTelemetry_RuntimeMetricsConfigFieldDefaultsOff(t *testing.T) { + t.Parallel() + + cfg := TelemetryConfig{ + LibraryName: "test-lib", + EnableTelemetry: false, + Logger: log.NewNop(), + } + assert.False(t, cfg.EnableRuntimeMetrics, + "EnableRuntimeMetrics must default to false (zero value)") +}