Align span/transaction type/subtype for OTel intake with agent bridge spec

[SylvainJuge]

Agent OTel bridge specification PR : elastic/apm#516

APM Server version (apm-server version): master as of writing, might be shipped into 7.16.0

For OTel intake, APM Server has to perform mapping from OTel attributes to the APM data model in ES.

When implementing an OTel bridge, agents have to replicate part of this mapping for the following fields:

• transaction.type
• span.type
• span.subtype
• span.destination.service.resource

Currently, in apm-server:

• when unable to infer transaction.type, it is set to custom (see code)
• when unable to infer span.type and span.subtype, it set to app / ${component}, where ${component} is provided by a component attribute in OTel intake. (see code)

---

There are multiple issues with this:

• On agent side, span.type fallbacks to custom when not explicitly set
• There is no component in the OTel specification 1.0.0
• Using custom as fallback for {span,transaction}.type could be confusing as this behavior is linked to usage of Agent API
• We have created a dedicated span type/subtype couple for internal spans : app/internal (currently in PR), this could be used for OTel spans whose kind is INTERNAL.

Within agent bridge specification, this mapping is described using gherkin specification

Bonus points if we can find a way to test apm-server with agent gherkin specs, we can easily add tags to filter out only the two relevant scenarios and their examples.

---

There are two options to change the current behavior:

Option 1: use introduce unknown type for fallback

• make the apm-server use unknown as fallback for both transactions and spans
• [trivial] this requires to add unknown as an allowed type in span type/subtype spec

Option 2: use custom as fallback

• make the apm-server use custom as fallback for spans, do not set span.subtype

Common

• fallback to app/internal for spans whose type/subtype can´t be inferred when OTel span kind is INTERNAL.

---

I have a slight preference for Option 1 as:

• it provides a clear distinction between usage of the Agent API that produces spans and transactions with custom type, hence making it clear that it might be the user/developer responsibility to change this default.
• with unknown, it makes it clear that the automatic mapping can't be applied, and that we (as Elastic) might have something to fix in the mapping.

[SylvainJuge]

@axw / @elastic/apm-server do you have any opinion on which option would be better ?
As OTel agent bridges would likely be shipped as experimental features, it might not be a big issue to have a small inconsistency until GA.

[axw]

@SylvainJuge I'm a bit confused by a couple of things:

On agent side, span.type fallbacks to custom when not explicitly set

I don't see any mention of "custom" in the Gherkin spec? Is this how things are now in agents, and you're proposing that we stop doing that?

[trivial] this requires to add unknown as an allowed type in span type/subtype spec

That seems to be already the case in the spec. Am I misreading it?

Option 1 seems sensible to me:

• default to app.internal for internal spans
• default to unknown for non-internal spans
• default to unknown for transactions
• remove the special ${component} attribute handling

[SylvainJuge]

Sorry for the confusion:

• I changed the agent spec PR to use Option 1 as we can see in the current state of the agent gherkin spec, this was before opening this issue.
• The current span type/subtype spec does not yet include unknown as an allowed span type, but apm#513 will add this bundled with few other changes.

Thus if we all agree on Option 1, it "only" requires to:

• merge apm#513 on agent side
• change implementation in apm-server

[axw]

@SylvainJuge thanks for clarifying. SGTM!