is otel telemetry support limited to grpc and how do I filter spans

[garypen]

Creating this on behalf of @talr1 in relation to issue: #2712

+++ COPY +++
I've tried adding otel telemetry (was the whole point of upgrading to 1.11.0 due to fixes in otel)

http did not work, grpc works fine. any idea?
working:
telemetry: tracing: trace_config: service_name: "federation-router" otlp: endpoint: "http://example-opentelemetry-collector:4317" protocol: grpc
not working:
telemetry: tracing: trace_config: service_name: "federation-router" otlp: endpoint: http://example-opentelemetry-collector:4318 protocol: http

any way to limit to amount of spans on each request? there seems to be ~8-9 spans per request. most of the time they don't add much useful info besides taking up storage?
spans names are:
request, router, supergraph, query_planning, parse_query, execution, fetch,
sub_graph, sub_graph_request

from my testing it seems that the amount of spans are tied to the router log_level (APOLLO_ROUTER_LOG). with level 'info' i can see ~8-9 spans as listed above. with level 'warn' no spans are emitted. I haven't checked other levels.

IMO: there should be at least 1 span on each service when i look on a full trace regardless of the service log level. in fact tracing is configured on the router but none is emitted in warn level.

Thanks in advance for any help,
+++ COPY +++

[ntkoopman]

For http you need to explicitly specify the tracing endpoint:

telemetry:
  exporters:
    tracing:
      otlp:
        enabled: true
        endpoint: http://traces.observability.internal:4318/v1/traces
        protocol: http

[bnjjj]

http protocol for metrics is not yet supported FYI.