[OpenTelemetry] Map OpenTelemetry Semantic Attributes to Elastic Common Schema when ingesting OTLP data

[cyrille-leclerc]

Map OpenTelemetry Semantic Attributes to Elastic Common Schema when ingesting OTLP data.

We should ensure that we map most of the OpenTelemetry semantic attributes to Elastic Common Schema.
We should prioritize attributes that are critical for

• The APM service map,
• Resource correlation,
• The visualisation of Otel data on Elastic APM screens

An important outcome will be to stop having Otel Attributes prefixed by labels. in our GUIs.

Examples

Otel Semantic Convention	Elastic ECS
container.image.name	container.image.name
k8s.cluster.name	?
k8s.pod.name	?
db.name	?
db.redis.database_index	?

Resources:

• OpenTelemetry Semantic Conventions specifications:
  • https://github.com/open-telemetry/opentelemetry-specification/tree/main/semantic_conventions/trace
  • https://github.com/open-telemetry/opentelemetry-specification/tree/main/semantic_conventions/resource
• Script used by the Otel Java SDK to generate constant files from the semantic conventions definitions: https://github.com/open-telemetry/opentelemetry-java/blob/main/buildscripts/semantic-convention/generate.sh
• Elastic Common Schema: https://www.elastic.co/guide/en/ecs/current/index.html

Example of documentation of the mapping for Jaeger: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk_exporters/jaeger.md

Example of attributes that are not mapped yet

Otel Attribute Ingested by Elastic	Sample value
labels.process_runtime_description	AdoptOpenJDK OpenJDK 64-Bit Server VM 15.0.2+7
labels.os_type	DARWIN
labels.process_executable_path	/Library/Java/JavaVirtualMachines/adoptopenjdk-15.jdk/Contents/Home:bin:java
labels.process_runtime_name	OpenJDK Runtime Environment
labels.os_description	Mac OS X 10.16
labels.service_namespace	com-shoppingcart
labels.process_command_line	/Library/Java/JavaVirtualMachines/adoptopenjdk-15.jdk/Contents/Home:bin:java -javaagent:./../.otel/opentelemetry-javaagent-all-0.17.0.jar -Dio.opentelemetry.auto.slf4j.simpleLogger.defaultLogLevel=info
labels.process_runtime_version	15.0.2+7
labels.telemetry_auto_version	0.17.0

{
  "_index": "apm-7.12.0-metric-000001",
  "_type": "_doc",
  "_id": "ug4U_ncBlMla0ZqF00Vo",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "opentelemetry/java",
      "version": "0.17.0"
    },
    "process": {
      "pid": 4990
    },
    "runtime": {
      "jvm": {
        "memory": {
          "area": 37609512
        }
      }
    },
    "processor": {
      "name": "metric",
      "event": "metric"
    },
    "labels": {
      "process_runtime_description": "AdoptOpenJDK OpenJDK 64-Bit Server VM 15.0.2+7",
      "area": "non_heap",
      "os_type": "DARWIN",
      "process_executable_path": "/Library/Java/JavaVirtualMachines/adoptopenjdk-15.jdk/Contents/Home:bin:java",
      "process_runtime_name": "OpenJDK Runtime Environment",
      "os_description": "Mac OS X 10.16",
      "service_namespace": "com-shoppingcart",
      "process_command_line": "/Library/Java/JavaVirtualMachines/adoptopenjdk-15.jdk/Contents/Home:bin:java -javaagent:./../.otel/opentelemetry-javaagent-all-0.17.0.jar -Dio.opentelemetry.auto.slf4j.simpleLogger.defaultLogLevel=info",
      "type": "used",
      "process_runtime_version": "15.0.2+7",
      "telemetry_auto_version": "0.17.0"
    },
    "observer": {
      "hostname": "MacBook-Pro.localdomain",
      "id": "f9c7166f-0f12-4fc4-8c1d-3c137d9b382a",
      "type": "apm-server",
      "ephemeral_id": "6e8d9a14-4392-49f5-b73a-e2df869bfc33",
      "version": "7.12.0",
      "version_major": 7
    },
    "@timestamp": "2021-03-04T16:31:52.663Z",
    "ecs": {
      "version": "1.6.0"
    },
    "service": {
      "environment": "staging",
      "name": "monitorOtlp",
      "language": {
        "name": "java"
      },
      "version": "1.0-SNAPSHOT"
    },
    "event": {
      "ingested": "2021-03-04T16:31:53.702349Z"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-03-04T16:31:53.702Z"
    ],
    "@timestamp": [
      "2021-03-04T16:31:52.663Z"
    ]
  },
  "sort": [
    1614875512663
  ]
}

[axw]

Here's a start, focusing on resource conventions.

OpenTelemetry os.* conventions:

OpenTelemetry	Elastic
os.type	os.type, os.platform
os.description	os.full

OpenTelemetry process.* conventions:

OpenTelemetry	Elastic
process.pid	process.pid
process.executable.name	?
process.executable.path	process.executable
process.command	?
process.command_line	process.command_line
process.command_args	process.args, process.args_count
process.owner	?
process.runtime.name	service.runtime.name (non-ECS, APM-specific keyword field)
process.runtime.version	service.runtime.version (non-ECS, APM-specific keyword field)
process.runtime.description	?

OpenTelemetry deployment.* conventions:

OpenTelemetry	Elastic
deployment.environment	service.environment (ECS RFC / draft, established in APM)

OpenTelemetry k8s.* conventions:

OpenTelemetry	Elastic
k8s.cluster.name	?
k8s.node.name	kubernets.node.name (non-ECS, produced by Beats and APM)
k8s.node.uid	?
k8s.namespace.name	kubernetes.namespace (non-ECS, produced by Beats and APM)
k8s.pod.uid	kubernetes.pod.uid (non-ECS, produced by Beats and APM)
k8s.pod.name	kubernetes.pod.name (non-ECS, produced by Beats and APM)
k8s.container.name	kubernetes.container.name (non-ECS, produced by Beats)
k8s.replicaset.uid	?
k8s.replicaset.name	kubernetes.replicaset.name (non-ECS, produced by Beats)
k8s.deployment.uid	?
k8s.deployment.name	kubernetes.deployment.name (non-ECS, produced by Beats)
k8s.statefulset.uid	?
k8s.statefulset.name	kubernetes.statefulset.name (non-ECS, produced by Beats)
k8s.daemonset.uid	?
k8s.daemonset.name	?
k8s.job.uid	?
k8s.job.name	?
k8s.cronjob.uid	?
k8s.cronjob.name	?

OpenTelemetry container.* conventions:

OpenTelemetry	Elastic
container.name	container.name
container.id	container.id
container.runtime	container.runtime
container.image.name	container.image.name
container.image.tag	container.image.tag

OpenTelemetry host.* conventions:

OpenTelemetry	Elastic
host.id	host.id
host.name	host.hostname, host.name
host.type	host.type
host.arch	host.architecture
host.image.name	?
host.image.id	?
host.image.version	?

OpenTelemetry cloud.* conventions:

OpenTelemetry	Elastic
cloud.provider	cloud.provider
cloud.account.id	cloud.account.id
cloud.region	cloud.region
cloud.zone, cloud.availability_zone	cloud.availability_zone
cloud.infrastructure_service	cloud.service.name (ECS next)

[axw]

Responding to #4919 (comment):

@cyrille-leclerc

A 25th hour in each day would be a good start ;)

@axw I wanted to understand if additional specification work was needed :-)

Sorry for the flippancy, I just meant that we have a lot going on, and I don't really have the time to dedicate beyond plugging the obvious gaps like those described in #4714 (comment).

For 7.13 I think we can manage to translate the attributes for which there are existing analogues in our agents, e.g. process_executable_path, process_runtime_name, process_runtime_version. Other attributes may have to wait. Let's discuss further on that issue if needed.

Does it mean that we/I should prepare this mapping?

If you could take a look at #4714 (comment) and let me know what you think that would be great. Any remaining gaps you can fill in would be much appreciated.

Should we align with @urso to have the same mapping used in the logs exporter for Elasticsearch and in the OTLP intake we have for traces and metrics?

I think we should certainly align as much as we can, to ensure we can correlate logs, metrics, and traces. I don't have the bandwidth to write a more formal spec/mapping right now. If either of you have time to take the lead on it, then I'm happy to contribute if and how I can.

[ebeahan]

There's an active discussion about introducing an orchestrator.* fieldset into ECS to capture container orchestrators' attributes, like k8s: elastic/ecs#1299.

Calling that to attention here since there might be an opportunity to leverage those proposed fields for the OTel k8s.* => ECS mappings.

[cyrille-leclerc]

FYI @AlexanderWert is working at clarifying and enhancing Elastic APM attributes for DB spans. It can relate to the work done here on the OpenTelemetry db.* attributes. See elastic/apm#420

[cyrille-leclerc]

@axw I feel it's hard to understand the status/completion of this ticket. Would it make sense to close it and to create finer grained tickets when gaps are identified?

[axw]

@cyrille-leclerc yes, SGTM