Need some help with tracing. The router is not generating any trace id, and i am not able to publish traces to jaeger either.

[vamshiaruru-virgodesigns]

Hi all, really need some help with tracing. I am moving from federation to router, and in federation Instrumentation took care of generating trace ids and propagating them. But in the router, I am not able to see any trace ids or publish traces to jaeger. Here's my config

supergraph:
  listen: 0.0.0.0:4006
  path: /graphql
  introspection: true

csrf:
  unsafe_disabled: true

sandbox:
  enabled: true

homepage:
  enabled: false

cors:
  match_origins:
    - ".*"
  allow_credentials: true

headers:
  all:
    request:
      - propagate:
          matching: .*

include_subgraph_errors:
  all: true # Propagate errors from all subraphs

traffic_shaping:
  deduplicate_variables: true
  router:
    timeout: 180s
  all:
    deduplicate_query: true
    timeout: 180s

plugins:
  ameyam.logger:
  ameyam.authentication:

telemetry:
  tracing:
    propagation:
      trace_context: true
    trace_config:
      service_name: ameyam_graphql_gateway
      sampler: always_on
    otlp:
      endpoint: default
    jaeger:
      collector:
        endpoint: http://localhost:14268

I have jaeger-all-in-one docker container up. I made a bunch of requests, and I couldn't see any traces in the jaeger UI.

I also inspected the headers that were being sent to my subgraphs, and I couldn't see open telemetry related headers anywhere. This is a big blocker for me to shift to router. We use trace ID to associate request in different subgraphs together (our subgraphs can read trace id from headers and append them to logs). Am I doing something wrong here? Any help is really appreciated! Thanks.

[vamshiaruru-virgodesigns]

I have tried inspecting trace ID using

                let trace = TraceId::maybe_new().map(|t| t.to_string());

But the trace is None :(

[bnjjj]

Oh ! That makes sense now. I think you're overriding the default global subscriber, we already set our own one and you can't override it. Because of this it takes only your own subscriber and not ours. Then you only have your features...
What do you need or you want to achieve with your own tracing configuration ?

[vamshiaruru-virgodesigns]

Mostly to log to a file instead of console, and use json formatting. In my previous experience, logging to console has performance implications and I was trying to avoid that.
And you're correct about overriding the default global subscriber. The .init() sets the global default. Is there a way to set the tracing configuration without overriding global config?

[bnjjj]

Why not using a unix redirection to a file ? And about json format you can set it from the config file https://www.apollographql.com/docs/router/configuration/logging#basic-configuration but if you're redirecting to a file it's automatically set to json if you don't specify it.

Is it something that seems convenient to you ?

[vamshiaruru-virgodesigns]

Yes, I have no issues with redirecting to a log file :) I am just concerned about the performance impact. I'll do some load testing and figure out if it works for me and get back to you :)

[vamshiaruru-virgodesigns]

Looks like things are fine with redirection. So I'll be going ahead with that.
Can't thank you enough for your help! I was stuck on this for a while and didn't realise the tracing was overriding router's tracing.