#[instrument(err)] causes broken traces in Datadog

[software-artificer]

I've been sent here after having a conversation in CNCF Slack in OTel Rust group. The issue tokio-rs/tracing#2648 was reported before, but it doesn't seem like that's a correct place for it.

Bug Report

We are using the tracing framework for our Rust applications for distributed tracing and sending traces and spans to Datadog using OTLP+gRPC.

When using #[tracing::instrument(err)] macro to capture and record Result::Err() returned by the annotated function, traces end up being broken in Datadog, reporting: Missing error message and stack trace.

I am willing to work through this issue and contribute a fix, provided some guidance as to where do I start from.

Please let me know if you need any additional information here.

Version

tracing v0.1.43
tracing-attributes v0.1.31 (proc-macro)
tracing-core v0.1.35
tracing-opentelemetry v0.32.0
tracing-subscriber v0.3.22

Platform

Google's Distroless container images based on Debian 12, x86_64.

Description

Here's the minimal reproducible example:

use opentelemetry::trace::TracerProvider as _;
use opentelemetry_otlp::WithExportConfig as _;
use tracing_subscriber::{layer::SubscriberExt as _, util::SubscriberInitExt as _};

#[tracing::instrument("Failing span", err)]
fn failing_function() -> Result<(), std::io::Error> {
    Err(std::io::Error::other("Big error, much problem"))
}

#[tokio::main]
async fn main() {
    let span_exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_tonic()
        .with_protocol(opentelemetry_otlp::Protocol::Grpc)
        // Update to point to your collector
        .with_endpoint("grpc://127.0.0.1:4317")
        .with_timeout(std::time::Duration::from_secs(5))
        .build()
        .expect("Failed to build a span exporter");

    let tracer_provider = opentelemetry_sdk::trace::SdkTracerProvider::builder()
        .with_batch_exporter(span_exporter)
        .build();

    let traces_layer =
        tracing_opentelemetry::OpenTelemetryLayer::new(tracer_provider.tracer("tracing-repro"));

    tracing_subscriber::registry()
        .with(traces_layer)
        .try_init()
        .expect("Failed to initialize tracing");

    failing_function().expect_err("Expected failing function to fail");

    tracer_provider
        .shutdown_with_timeout(std::time::Duration::from_secs(5))
        .expect("Failed to shutdown the tracer provider");
}

and the Cargo.toml to go with it:

[package]
name = "tracing-instrument-otel-err"
version = "0.1.0"
edition = "2024"

[dependencies]
opentelemetry = { version = "0.31.0", default-features = false, features = ["trace"] }
opentelemetry-otlp = { version = "0.31.0", default-features = false, features = ["grpc-tonic", "trace"] }
opentelemetry_sdk = { version = "0.31.0", default-features = false, features = ["trace"] }
tokio = { version = "1.48.0", default-features = false, features = [
	"macros",
	"rt-multi-thread",
] }
tracing = { version = "0.1.43", default-features = false, features = ["attributes", "std"] }
tracing-opentelemetry = { version = "0.32.0", default-features = false }
tracing-subscriber = { version = "0.3.22", default-features = false, features = ["std", "registry"] }

When using this code and sending traces to Datadog, the following problem can be observed:

and the error tab, respectively:

[mladedav]

The older linked issue seems to be somewhat obsolete as it says that there should be a field that's been since deprecated.

This honestly seems more like a data dog issue but if we can fix it by aligning better with semantic conventions, we can do that.

Can you try manually adding fields into your spans that you think are the ones data dog is kissing to pinpoint what exactly it expects us to send?

[cijothomas]

It does produce a Span with failed status, and an event with "exception" as its name, and have "exception.message" attribute. This is what OTel convention has as well.
https://github.com/open-telemetry/semantic-conventions/blob/main/docs/exceptions/exceptions-spans.md

    Name         : Failing span
    TraceId      : 4c395809765a7b11b0c6e821e1d3e9f8
    SpanId       : adca475c0474f4ae
    TraceFlags   : TraceFlags(1)
    ParentSpanId : None (root span)
    Kind         : Internal
    Start time   : 2025-12-04 21:33:09.304219
    End time     : 2025-12-04 21:33:09.304508
    Status       : Error { description: "Big error, much problem" }
    Attributes:
             ->  code.file.path: String(Static("examples/basic.rs"))
             ->  code.module.name: String(Static("basic"))
             ->  code.line.number: I64(8)
             ->  thread.id: I64(1)
             ->  thread.name: String(Owned("main"))
             ->  target: String(Static("basic"))
             ->  busy_ns: I64(164708)
             ->  idle_ns: I64(142167)
    Events:
    Event #0
    Name      : exception
    Timestamp : 2025-12-04 21:33:09.304499
    Attributes:
             ->  level: String(Static("ERROR"))
             ->  target: String(Static("basic"))
             ->  exception.message: String(Owned("Big error, much problem"))
             ->  code.file.path: String(Static("examples/basic.rs"))
             ->  code.module.name: String(Static("basic"))
             ->  code.line.number: I64(8)

[mladedav]

Maybe it expects the stacktrace attribute. Which is just recommended and I think that capturing backtraces is disabled by default.

[software-artificer]

I have managed to get things working. Datadog expects exception.type to be present:

PR's incoming!

[mladedav]

According to the semantic conventions, the type field is required only if there's no message and recommended otherwise.

The problem with this is, if someone writes

let result: std::io::Result<()> = fallible_operation();
if let Err(inner_error) = result {
  tracing::error!(error = ?inner_error, "Things went sideways");
}

we'll get exception.type = &dyn Debug which is not helpful at all. We can't get to the actual type and I assume this is the exact scenario why the spec doesn't require the field unconditionally.

---

I haven't found an issue about this but maybe datadog could handle the missing fields. Otherwise we can add a placeholder there but maybe I'd just put there unknown or something like that rather than the more specific yet probably misleading &dyn Debug and friends since all errors also implement Debug and it's notably hard to get &dyn Error into tracing.

[software-artificer]

@mladedav I wholeheartedly agree with your argument here. I pushed an extra commit to update the exception.type to Unknown in my PR. If no objections, I would like to proceed with this approach to unblock things, but I am definitely not satisfied with things as is and would like to figure out some better solutions if possible.

It is unfortunate that due to the type erasure and use of dyn we can't recover the original type here. However, this got me thinking: provided most of the interactions with tracing are done by means of macroses, could we try and capture, at least in some cases like tracing::instrument, the types we are dealing with? I get it, this will require changes to the tracing itself, but it might not be that bad of an idea.

Or perhaps even have an implementation that takes e: &T where T: std::error::Error instead of &dyn std::error::Error that would allow us to recover the type of T via std::any::type_name::<T>() for special cases of errors?

[Mirko-von-Leipzig]

I imagine everyone else would be slightly horrified at having a concrete tracing::instrument impl for every error type suddenly be generated. Maybe a new Error: std::error::Error which adds type information (optionally)? This would allow the dyn Error to remain at least, and also let error types override the type name if its ambiguous or clunky.

Have you considered a custom instrument macro that does it this way to try it out? I've been flirting with doing something like this to improve on the tracing defaults e.g. having skip_all as default, and an error report format in addition to % ?.

[software-artificer]

@Mirko-von-Leipzig it is a valid point. My hope is that all of this can largely remain resolved at compile time anyway, so would people care that much? I can't tell, maybe they will. Would you personally be concerned about this? I also wouldn't expect this to generate the impl for every error type, only ones that are instrumented, which does not mean "all of them".

I certainly can create and maintain custom macros for this, but I would like the out of the box experience for everyone to be great, so I hope we can at least give it a shot and see if something better can be achieved. If all of this falls through, then sure, this will be my last resort.

[Mirko-von-Leipzig]

@Mirko-von-Leipzig it is a valid point. My hope is that all of this can largely remain resolved at compile time anyway, so would people care that much? I can't tell, maybe they will. Would you personally be concerned about this? I also wouldn't expect this to generate the impl for every error type, only ones that are instrumented, which does not mean "all of them".

The problem is that this will generate a concrete implementation for all unique T that are instrumented in all of your dependencies. And this code implementation is delayed until the generic is invoked (so to speak) which means it massively bulks up compile time and binary size.

Although this does depend on how this is implemented internally; so maybe it is this way already since its a macro.

[mladedav]

Tracing is a bit short-staffed so I wouldn't expect large changes like this to land anytime soon.

Anyway, when you want to pass something as a field to tracing, it needs to implement Value. You can have only one of these three at the same time: impl Value for &str, impl Value for T where T: Debug, and impl Value for T where T: Error. Using trait objects sidesteps that.

Feel free to explore the solution space more, maybe you'll find some better way, but I don't think there is a non-breaking change to do what you want. I think valuable might help.

[software-artificer]

Thank you @mladedav for your continued support here! I went looking into Value implementations and there are quite a few, but when it comes to macros usages, tracing::instrument(err) ends up sending the DisplayValue<T>, not even a proper std::error::Error.

It is of course possible to capture the actual type via std::any::type_name() in the Value implementations, but we will need to propagate the captured type to the visitor (which will likely be a breaking change) and I'm not feeling particularly great about using std::any in something like tracing which might be used in performance critical path. Plus this will capture types for all fields, not just error scenarios, which feels wasteful to me.

With the project being short-staffed, what is the best way to move things like this forward? I lack the proper knowledge about tracing internals and need guidance / feedback, but happy to continue to poke at things. Is there a better medium (Discord maybe?) to help brainstorm options?

[mladedav]

It is of course possible to capture the actual type via std::any::type_name() in the Value implementations

If you have impl Value for T where... then yes, if you have impl Value for dyn Debug then no.

You can try the tracing-internals channel on Discord and you can open an issue in the tracing repo (the former is more likely to get some opinions, the latter is better if you want it tracked long-term). But 0.2 is probably not going to happen anytime soon and breaking changes aren't being merged at this time I think.

[djc]

@bantonsson do you think it might make sense for Datadog to relax its requirements here?

[bantonsson]

do you think it might make sense for Datadog to relax its requirements here?

@djc Absolutely, I'll try to find someone who knows if this is only UI/query related or affects more parts of the processing.

@software-artificer are you sending this data directly to Datadog, or via the Datadog Agent or an OpenTelemetry Collector? I'm just wondering for completeness. Also, do you have a support ticket open with Datadog for this? It would be great to be able to reference it internally.