`IncludedData` option to flatten structures into dotted properties for receivers without `Map` support

[justastudenttryingtolearn]

Description

When logging complex objects using Serilog and sending them to Elastic via Serilog.Sinks.OpenTelemetry, the structure of the object is not preserved.
Instead, the entire object is serialized into the message field as a string, rather than being mapped to individual fields in Elastic.
Logging directly to Elastic using Serilog works as expected.

Reproduction

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.OpenTelemetry(options =>
    {
        options.Protocol = OtlpProtocol.HttpProtobuf;
    })
    .CreateLogger();

var quote = new Quote
{
    Id = 1,
    Author = "Albert Einstein",
    Text = "Life is like riding a bicycle. To keep your balance you must keep moving.",
    CreatedAt = DateTime.UtcNow,
    Tags = new List<string> { "life", "balance", "movement" },
    Metadata = new Dictionary<string, object>
    {
        { "source", "BrainyQuote" },
        { "rating", 5 },
        { "isFavorite", true }
    }
};

Log.Information("Logging a complex object: {@Quote}", quote);

Minimal repo for reproduction:
https://github.com/justastudenttryingtolearn/TestingSerilogSinks/tree/main/TestingSerilogSinks

Elastic Output (via OTEL Sink):

{
  "message": "Logging a complex object: {\"Id\":1,\"Author\":\"Albert Einstein\",\"Text\":\"Life is like riding a bicycle. To keep your balance you must keep moving.\",\"CreatedAt\":\"2025-08-22T11:40:51.2619620Z\",\"Tags\":[\"life\",\"balance\",\"movement\"],\"Metadata\":{\"source\":\"BrainyQuote\",\"rating\":5,\"isFavorite\":true},\"$type\":\"Quote\"}"
}

Expected behavior

Each property in the log event should be mapped to an OpenTelemetry attribute, and then to an Elastic field with the same name.
The value's structure (e.g., nested objects, arrays) should be maintained all the way to Elastic, so that each property is available as a separate field.

Relevant package, tooling and runtime versions

• Serilog version: 4.3.1-dev-02373
• Serilog.Sinks.OpenTelemetry version: 4.2.1-dev-02306
• .NET version: 8.0.408

Additional context

Ideally, each property of the complex object should be available as a separate field in Elastic, preserving the structure.
Please advise if this is a limitation of the OTEL sink or if there is a recommended workaround.

[nblumhardt]

Hi! Structured data is fully supported, you might need to dig in deeper to figure out where the Quote property ends up.

The quote will still be JSON-serialized into the message, as you've shown, but there should be a top-level Quote property alongside it that carries the structured representation of the object.

It's possible Elasticsearch doesn't support structured data in OTLP property values, and is dropping the property off.

The best way to debug would be to clone this repo, put some breakpoints in the exporter, and check what the GRPC payload looks like. This will narrow the problem down to being on this side, or on the Elastic side. HTH!

[justastudenttryingtolearn]

Thanks for the detailed reply. I've cloned the repo and subsequently run the "example project" as per the README, only changing it so that it used LogContext.Pushproperty to log (as that is what we do in our project). I’ve attached a snippet from my collector log output (log.txt) for reference:
 

LogRecord #0
Body: Str(The position is {"Latitude":-43,"Longitude":19})
Attributes:
     -> Position: Map({"Latitude":-43,"Longitude":19})
     -> Protocol: Str(http/protobuf)
     -> Elapsed: Int(4)
     -> message_template.text: Str(The position is {@Position})

 
As you can see, the Position property is mapped as a single attribute with a map value, but its structure is not fully destructured into individual fields. Nested properties (like Latitude and Longitude) are not promoted to top-level attributes, so Elastic only receives the object as a whole, not its individual properties.

Is there a possible work around to force complex objects to be destructured into top level objects?

 
Otherwise we will keep looking into it

[nblumhardt]

Hi! Thanks for the follow-up.

The sink is working correctly here, and preserving all the structure in that Map object. Splitting that into individual fields is reasonable if Elastic requires it, but it's a lower-fidelity representation we'd only enable for compatibility purposes.

There's a mechanism in this sink to control how data is mapped into OTLP payloads called IncludedData; I'm not sure how much effort and churn would be involved in adding an IncludedData.FlattenedStructureValues (or similar) but it seems reasonable. I'll re-title the issue accordingly.

[justastudenttryingtolearn]

Hi! Thanks so much for your help and the detailed explanation. I really appreciate it. If you need any assistance with testing IncludedData.FlattenedStructureValues or anything else, I'd be happy to help - just let me know!

[ReinhardvdBerg]

This feature will be very useful, looking for similar attribute flattening to what can be done with the dotnet logger here:
https://opentelemetry.io/docs/languages/dotnet/logs/complex-objects/
Thank you