feat(telemetry): add custom headers for OTLP exports (metrics, tracing, access logs) #7674
Conversation
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
from
63fd03f to
04c7b52
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7674 +/- ##
==========================================
+ Coverage 72.34% 72.41% +0.07%
==========================================
Files 234 235 +1
Lines 34542 34566 +24
==========================================
+ Hits 24988 25032 +44
+ Misses 7762 7746 -16
+ Partials 1792 1788 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
2 times, most recently
from
c58e24a to
5f69a4f
Compare
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
2 times, most recently
from
2483048 to
36f5bee
Compare
|
ok made a revision also to PR desc to hopefully answer questions in comments. I will revisit this again monday if there is more feedback. |
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
from
36f5bee to
1a16872
Compare
|
pulling into draft while I remove the port dodging things for things we can't control yet. |
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
from
1a16872 to
47998d6
Compare
|
also I just realized we never added this for tracing either, so will add that to the PR |
codefromthecrypt
changed the title
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
from
a5b5a19 to
c7e3ecf
Compare
| spec: | ||
| telemetry: | ||
| metrics: | ||
| # TODO: Some backends like otel-tui and Elastic require delta temporality |
There was a problem hiding this comment.
These problems are too much to do in this PR, and I can over time try to clean these up also
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
3 times, most recently
from
607296e to
5bff38d
Compare
| log := collector.TakeLog() | ||
| require.NotNil(t, log) | ||
| require.Contains(t, log.Body.GetStringValue(), `HTTP/1.1" 200`) | ||
| require.Equal(t, "Bearer test-api-key", testotel.GetAttributeString(log.Attributes, "grpc.metadata.authorization")) |
There was a problem hiding this comment.
@mathetake @anuraaga you might like this sneaky trick to verify the headers/metadata received by otel. propagate them into fake span attrs
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
2 times, most recently
from
412773b to
ea11614
Compare
|
ok I updated this as I noticed the same thing was missing everywhere, in logs metrics and tracing. fixed so that the tests are coherent and we don't need to go back and clean this up multiple times. There's been a history of fragmentation and not following up later, so doing otel holistically for one thing is a good thing. |
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
4 times, most recently
from
91a89a9 to
f02a62e
Compare
|
will undraft after #7697 because I noticed metrics are unreadable in otel-tui until that change (many otel backends require delta temporality) |
…sink This enables OTLP access logs to have both body (text) and attributes populated simultaneously, matching the OpenTelemetry LogRecord spec. Previously, the GatewayAPI translator used a switch statement that forced choosing EITHER body (from format.Text) OR attributes (from format.JSON), never both. This did not match the OTLP spec where body and attributes serve complementary purposes: - body: The primary log message - attributes: Structured metadata for querying/filtering The new sink-level text and attributes fields allow configuring both directly, with fallback to setting-level format for backwards compatibility. Signed-off-by: Adrian Cole <[email protected]>
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
from
f02a62e to
59670b7
Compare
|
@arkodg @zirain I looked carefully at my screenshots and noticed that the logs didn't include any otel attributes. this is a bug, even if very small one. I raised #7720 as we shouldn't merge this until it works like one would expect (the example should be valid and actually have attributes in otel for the log). once that is merged I think we're finally good. thanks for your patience |
|
updated the screenshots based on the current branch which cherry-picks and shows the attributes for the log messages get to otel |
Adds a Headers field to the OpenTelemetry configuration for access logging, tracing, and metrics. This allows configuring gRPC initial metadata (e.g., Authorization headers) for authenticated OTLP export to collectors like Elastic, Datadog, or cloud providers. Also adds MinItems=1 validation for OTLP headers. Signed-off-by: Adrian Cole <[email protected]>
codefromthecrypt
force-pushed
the
otel-grpc-headers
branch
from
59670b7 to
a7376ae
Compare
3 participants
What type of PR is this?
feat(telemetry): add new feature
What this PR does / why we need it:
Adds support for custom headers on all OTLP/gRPC exports (metrics, tracing, and access logs), enabling authentication with collectors like Elastic Cloud, Datadog, or cloud providers that require API keys or bearer tokens.
This uses Envoy's
GrpcService.initial_metadatato send headers as gRPC metadata.Release Notes: Yes
Notes:
This PR includes an example showing authenticated OTLP exports for all three signals. The example uses otel-tui, but could use anything else.
Changes
Headersfield toProxyOpenTelemetrySinkintelemetry.metrics.sinks[].openTelemetryHeadersfield toTracingProviderintelemetry.tracing.providerHeadersfield toOpenTelemetryAccessLogintelemetry.accessLog.settings[].sinks[].openTelemetryAll three use the same pattern: a list of
HTTPHeaderobjects withnameandvaluefields.FAQ
Why not use SecretObjectReference for sensitive headers?
For EnvoyProxy config that supports file-based standalone mode, Kubernetes secrets are not available. Plain text headers are needed regardless.
For example, Honeycomb requires non-secret headers like
x-honeycomb-dataset. Secret support can be added in a follow-up for Kubernetes mode.