Exclude resource from metric/measurements

[thompson-tomo]

What are you trying to achieve?

I want to be able to have mobile applications to be emitting metrics without resulting in high cardinality due to being associated with the resource.

What I am thinking would help is if on either the metric or data point we can indicate that it should not be associated with the resource hence resolving the cardinality problem.

Additional context.

There is discussion by the mobile/browser sig to avoid using metrics due to high cardinality.

_{Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[MrAlias]

How does associating a resource with a static set of attributes increase the cardinality of the metric set?

This seems categorically wrong. Further explanation of the problem is needed.

[thompson-tomo]

Because mobile applications are traditionally installed on significantly higher volumes of devices compared to web application back end. This qty of installs will result in large number of resources which is ok for traces however for metrics results in seperate buckets of data leading to cardinality scenarios 0.

[mx-psi]

There is discussion by the mobile/browser sig to avoid using metrics due to high cardinality.

For reference, this seems to be the issue where this is being discussed: #4604

[pellared]

@thompson-tomo, I think that in the mobile applications you can create a dedicated resource for your metrics SDK. I think you may still want to have some resource attributes (like OS type, OS version).

I strongly believe that excluding the resource is a bad idea.
You can always pass an empty resource if this is what you really want.

[thompson-tomo]

@mx-psi is correct that with one such example in particular the comment #4604 (comment)

@pellared i agree metrics could have a different resource however there is scenarios where in which the resource attributes ie os.version provides limited benefit. An example would be http client duration metrics, hence idea is to opt out of the resource association on a per metric/data point. On the other hand install base metrics that additional info about device/os is of benefit.

I am more than open to alternatives which solves the cardinality while removing the need for the guidance to not implement metrics.

Also in declarative config, it's not currently possible to define different resources for the different signal types.

[bidetofevil]

Cardinality is not the only issue with OTel metrics and mobile and web apps. It's that simply aggregating some countable thing for an entire device -- let alone the entire fleet -- is, most of time, not very useful. You want to know WHEN things like frame drops or data usage is happening (e.g. before or after some event) to establish correlation or causality. Having periodic collection of such metrics doesn't give you the necessary data points to do that unless your aggregation time window is really short, which complicates things even further.

Excluding resources has the opposite effect: you have a number for frame drops for an entire fleet, which won't even tell you what devices are affected. It may provide some value in cases where that doesn't matter, but it's not a core use case, so I don't think it warrants this exception.

[thompson-tomo]

Let me be more explicit about my previous comment which addresses @bidetofevil concern and was implicitly implied as it is standard functionality.

By excluding the resource yes you lose the automatic association with the device generating the metric however this does enable the metric to define what attributes associated with the device should be captured ie os.version, hence providing benefit across the board.

If you are interested in knowing when something happened then that by definition is an event hence is not relevant to the discussion here.

For metrics you also have examplars which enables you to associate a metric data point with a trace. This gets useful as you can now see the span which generated the metric data point.

[bidetofevil]

What are you hoping to achieve with this resource-less metrics collection? Specifically, what specific kinds of telemetry are you proposing to gather using this, and how would it be processed on the Collector side (e.g. aggregated, associated with other data points, etc.)?

If you are looking to add something that is shaped like an OTel metric data point to spans generically, I don't think doing through actual OTel metrics with the way the spec is defined is the right way to go. By definition, exemplars are special cases, example data points, and you'd basically need to make every data point an exemplar, which doesn't feel in the spirit of what they were designed for. I'll let the Metrics spec experts confirm or deny that last point.

Zooming out, it seems to me that creating instrumentation that periodically or on an as-needed basis emit events with known schemas containing the data you want to capture and associating them with the appropriate spans (or even without that) would be a better way to go. Metrics can then be gather on the Collector by going through the attributes of those events, and cardinality wouldn't be an issue.

Alternatively, if you are looking for an API similar to what OTel Metrics offers, but the signals emitted wouldn't necessarily be an OTel metric as the spec has defined it, especially in terms of how they are processed at the Collector layer, then you might be wanting something more involved than simply OTel metrics without an association with a Resource.

[thompson-tomo]

As mentioned previously an example would be http client duration metrics where knowing the device is not that relevant but knowing the app/app version is relevant.
You could also have metrics for errors/operations. Even without examplars if your metric is showing high error levels, you can then manually search for events which use the same attributes to get the details of scenarios.

I disagree with your assumption that every data point needs to be an examplar but that is auxiliary to the focus here.

What I want is to ensure that we have a way in which to support metrics from mobile which doesn't result in guidance to not use them while also not forcing users to implement transformations which could've already been done prior to transmission.

I fail to see how the idea of specifying that a metric should not be associated with the resource results in an outcome any different for a enduser of the telemetry to what the collector transformation would produce without needing to run a server side transformation.

[bidetofevil]

Not all metrics (lower case m) from OTel have to come to OTel Metrics (capital M). In fact, OTel Metrics is designed to send pre-aggregated times-series data. Per the spec:

The OpenTelemetry data model for metrics consists of a protocol specification and semantic conventions for delivery of pre-aggregated metric timeseries data

There doesn't seem to be any pre-aggregation happening here. And if you aggregate all the durations of HTTP requests on a specific device, then it goes to the problem I mentioned before about the data being not very useful.

For metrics about HTTP request duration, you can record it as a span per convention, and produced metrics on the Collector side by scanning through the appropriate spans, which can then be sent as OTel Metrics places like Prometheus.

In other words, you can use metrics in OTel without using OTel metrics. In fact, that's how all client side apps, mobile and web, do it. If your other use cases are like HTTP requests, then I think you can already do what you want via OTel, just not through capital M OTel Metrics.

[thompson-tomo]

Pre-aggregation would be occurring so no idea where that wrong assumption comes from.

As you bring up the spec, the use case for Mobile otel metrics is actually described:

1. OTel SDK exports 10 second resolution to a single OTel collector, using cumulative temporality for a stateful client, stateless server:

• Collector passes-through original data to an OTLP destination
• Collector re-aggregates into longer intervals without changing attributes
• Collector re-aggregates into several distinct views, each with a subset of the available attributes, outputs to the same destination

3. OTel SDK exports both 10 seconds resolution (e.g. CPU, request latency) and 15 minutes resolution (e.g. room temperature) to a single OTel Collector. The collector exports streams upstream with or without aggregation.

4. A number of OTel SDKs running locally each exports 10 second resolution, each reports to a single (local) OTel collector.

• Collector re-aggregates into 60 second resolution
• Collector re-aggregates to eliminate the identity of individual SDKs (e.g., distinct service.instance.id values)
• Collector outputs to an OTLP destination

Effectively my proposal would help with these by providing a flag on a per metric/data point to indicate where the identity should be removed and attributes should simply be used.

[cijothomas]

I want to be able to have mobile applications to be emitting metrics without resulting in high cardinality due to being associated with the resource.

Even if all resources are added at metric attributes, it won't change cardinality within the OTel SDK implementations, as Resource is static/fixed and multiplying 1 to existing cardinality won't increase it.

Also, there is nowhere in OTel SDKs that Resources are added automatically as Metric attributes. Resource information is available - but its up to user to specify which Resource attributes are to be promoted as Metric attributes.. This is done typically in OTel Collector, not in language SDKs..

[tsloughter]

basis emit events with known schemas containing the data you want to capture and associating them with the appropriate spans

This exactly. Otel has the concept of Events which are logs with a structure (schema). For measuring things like frame drops I think it sounds like it makes sense as Events with a span context attached (essentially what Span Events were but they are being deprecated for using Events with attached Context). I say "I think" because I'm basing this on what I've heard you say about what you'd like to measure and not from any experience with mobile development :)