OpenTelemetry vs Prometheus label names

[arendjr]

I noticed the draft spec contains both OpenTelemetry and Prometheus specifications for all metric names. Most notably, OpenTelemetry metric names use . as a separator, while Prometheus metric names use _. The Prometheus ones also sometimes have suffixes added.

The reason for this dual-naming is because Prometheus conventions are codified in the OpenMetrics standard. For anyone curious to dive into the rabbit hole, we've had a previous OpenTelemetry vs OpenMetrics discussion on this: https://github.com/orgs/autometrics-dev/discussions/37

Unfortunately, we haven't considered this situation with regards to labels yet. Labels use . as separator in the draft spec, which is great according to OpenTelemetry, but is forbidden according to the OpenMetrics ABNF. Our current clients also use underscores as separators AFAIK, so they're de-facto following OpenMetrics today.

Fortunately, my recommendation would be simple: We keep the Autometrics draft spec as is, but suggest that all dots (.) in label names should be replaced with underscores (_) when submitting to Prometheus.

[brettimus]

seems raisonable

[hatchan]

When you say "submitting to Prometheus", as you referring to the exposition format at the /metrics endpoints (and possibly the non otel push metrics to gravel/pushgateway)?

[arendjr]

Yes, exactly.

[hatchan]

How will the OTEL push flow be impacted by this? Will these payloads still contain the .?

[arendjr]

That's what I would expect, yes.

[hatchan]

I wonder what happens when you push a OTEL payload to a otel collector or directly to Prometheus ... Will it be translated 🤔 Maybe something that I can prototype tomorrow...

[gagbo]

I did this already, and the otel collector accepts the . without issue. I assume that the collector does the _ translation when sending to Prometheus

[arendjr]

I noticed while implementing this PR that at least the TypeScript OTEL libraries convert the dots back to underscores for the Prometheus exporter for us: autometrics-dev/autometrics-ts#140

So I think it's reasonable to expect OTLP => Prometheus conversions elsewhere to do this too.

[hatchan]

I can confirm what @gagbo said. The following flow works as expected (ie. no dots, just underscores): application -> otel collector -> remotewrite endpoint (prometheus).

[hatchan]

Would it maybe make sense to just avoid this whole issue, by not using the . altogether and use _ for everything? We do use some of the same names that are present in the OTEL spec, but I think these are quite loosly coupled.

[arendjr]

My understanding was that would go against the Otel conventions/recommentations. For instance, Otel explicitly uses service.name, whereas in Prometheus it would become service_name.

[hatchan]

After looking through the draft 1.0 spec I found that this is already part of it:

autometrics-shared/specs/autometrics_v1.0.0.md

Lines 75 to 77 in 7409f02

	When the metrics are exported to Prometheus, all dot (`.`) separators in the
	metric name and label keys are replaced by underscores (`_`). Suffixes are
	appended where required by Prometheus/OpenMetrics.