From 618173078749e3c081e484b10a2ad2fc2650e52b Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 25 Mar 2025 18:57:16 +0000 Subject: [PATCH 01/34] Start delta proposal Signed-off-by: Fiona Liao --- ...25-03-25_otel_delta_temporality_support.md | 192 ++++++++++++++++++ 1 file changed, 192 insertions(+) create mode 100644 proposals/2025-03-25_otel_delta_temporality_support.md diff --git a/proposals/2025-03-25_otel_delta_temporality_support.md b/proposals/2025-03-25_otel_delta_temporality_support.md new file mode 100644 index 0000000..2a9d872 --- /dev/null +++ b/proposals/2025-03-25_otel_delta_temporality_support.md @@ -0,0 +1,192 @@ + +## Your Proposal Title + +* **Owners:** + * @fionaliao +TODO: add others from delta wg + +* **Implementation Status:** `Not implemented` + +* **Related Issues and PRs:** + * `` + +* **Other docs or links:** + * `` + + This design doc proposes adding native delta support to Prometheus. This means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. + +## Why + +Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter-like OTEL metrics (e.g. histograms, sum) can have either [cumulative or delta temporality](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality). + +However, Prometheus only supports cumulative metrics, due to its pull-based approach to collecting metrics. + +Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167[). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. + +Example conversion: +``` +T0: 0 -> T0: 0 +T1: 1 -> T1: 1 +T2: 1 -> T2: 2 +T3: 5 -> T3: 7 +``` + +The benefit of the current solution is that it is simple - the cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. + +Prometheus' goal of becoming the best OTEL metrics backend means we should make sure to support delta metrics properly. We propose to add native support for OTEL delta metrics (i.e. metrics ingested via the OTLP endpoint). Native support means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. + +### Delta datapoints + +In the [OTEL spec](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality), like cumulative metrics, a datapoint for a delta metric has a `(start,end]` time window. However, the time windows of the previous datapoint and the current datapoints do not overlap. +The `end` timestamp is referred to as `TimeUnixNano` and is mandatory. The `start` timestamp is referred to as `StartTimeUnixNano`. This second timestamp is optional, but recommended for better rate calculations and to detect gaps and overlaps in a stream. Examples where `StartTimeUnixNano` is not added include the countconnector + +### Characteristics of delta metrics + +Sparse metrics are more common for delta than cumulative metrics. While delta datapoints can be emitted at a regular interval, in some cases (like the OTEL SDKs), datapoints are only emitted when there is a change (e.g. if tracking request count, only send a datapoint if the number of requests in the ingestion interval > 0). This can be beneficial for the metrics producer, reducing memory usage and network bandwidth. + +Further insights and discussions on delta metrics can be found in [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y), which describes Chronosphere's experience of adding functionality to ingest OTEL delta metrics and query them back with PromQL, and also [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q). + +### Pitfalls of the current solution + +#### Lack of out of order support + +Delta to cumulative conversion requires adding up older delta samples values with the current delta value to get the current cumulative value, so deltas that arrive out of order cannot be added without rewriting newer samples that were already ingested. + +As suggested in an [earlier delta doc](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q), a delay could be added to collect all the deltas within a certain time period before converting them to cumulative. This means a longer delay before metrics are queryable, so would not be suitable for long out of order time windows. + +#### No added value to conversion + +The benefit of cumulative metrics is that they’re resilient to data loss - if a sample is dropped, the next sample will still include the count from the previous sample. With delta metrics, if a sample is dropped, its data is just lost. Converting from delta to cumulative doesn’t improve resiliency as the data is already lost before it becomes a cumulative metric. + +Cumulative metrics are usually converted into deltas during querying (this is part of what rate() and increase() do), so converting deltas to cumulative is wasteful if they’re going to be converted back into deltas on read. + +#### Conversion is stateful + +Converting from delta to cumulative requires knowing previous values of the same series, so it’s stateful. Users may be unwilling to run stateful processes on the client-side (like the deltatocumulative processor). This is improved with doing the delta to cumulative conversion within the Prometheus OTLP endpoint, as that means there’s only one application that needs to maintain state (Prometheus is stateful anyway). + +State becomes more complex in distributed cases - if there are multiple OTEL collectors running, or data being replicated to multiple Prometheus instances. + +#### Values written aren’t the same as the values read +Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. call to get a useful result. However, it could be confusing, especially to newcomers to Prometheus, that when querying just the metric without any functions, the returned value is not the same as the ingested value. + +#### Does not handle sparse metrics well +As mentioned in Background, sparse metrics are more common with delta. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. There is more discussion on this in the Querying deltas section. + +## Goals + +Goals and use cases for the solution as proposed in [How](#how): + +• Allow OTEL delta metrics to be ingested via the OTLP endpoint and stored as-is +• Support for all OTEL metric types that can have delta temporality (sums, histograms, exponential histograms) +• Queries behave appropriately for delta metrics + +### Audience + +This document is for Prometheus server maintainers, PromQL maintainers, and Prometheus users interested in delta ingestion. + +## Non-Goals + +* Support for ingesting delta metrics via other means (e.g. remote-write) +* Support for non-monotonic sums + +These may come in later iterations of delta support, however. + +## How + +### Ingesting deltas + +#### Storing samples +When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. + +TODO: start time nano injection + +For the initial implementation, reuse existing chunk encodings. + +Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. If a value in a bucket drops, that counts as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Therefore a new counter reset hint/header is required, to indicate the cumulative counter reset behaviour for chunk cutting should not apply. + +##### Alternatives + +###### CreatedTimestamp per sample +If `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. The CreatedTimestamp field does not exist yet, but there is currently an effort towards adding it for cumulative counters ([PR](https://github.com/prometheus/prometheus/pull/16046/files)), and can be reused for deltas. Having the timestamp and the start timestamp stored in a sample means that there is the potential to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. + +###### Treat as gauge +To avoid introducing a new type, deltas could be represented as gauges instead and the start time ignored. + +This could be confusing as gauges are usually used for sampled data (for example, in OTEL: "Gauges do not provide an aggregation semantic, instead “last sample value” is used when performing operations like temporal alignment or adjusting resolution.”) rather than data that should be summed/rated over time. + +###### Treat as “mini-cumulative” +Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. + +This requires CT-per-sample to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. + +Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. rate() assumes samples in a range are equally spaced to figure out how much to extrapolate, which is less likely to be true for delta samples. + +This also does not work for samples missing StartTimeUnixNano. + +###### Convert to rate on ingest +Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixName` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. + +The difference is that there is no interval information in the metric name (like :rate1m) as there is no guarantee that the interval from sample to sample stays constant. + +To averages rates over more than the original collection interval, a new time-weighted average function is required to accommdate cases like the collection interval changing and having a query range which isn't a multiple of the interval. + +This would also require zero timestamp injection or CT-per-sample for better rate calculations. + +Users might want to convert back to original values (e.g. to sum the original values over time). It can be difficult to reconstruct the original value if the start timestamp is far away (as there are probably limits to how far we could look back). Having CT-per-sample would help in this case, as both the StartTimeUnixNano and the TimeUnixNano would be within the sample. However, in this case it is trivial to convert between the rated and unrated count, so there is no additional benefit of storing as the calculated rate. In that case, we should prefer to store the original value as that would cause less confusion to users who look at the stored values. + +This also does not work for samples missing StartTimeUnixNano. + +#### Distinguishing between delta and cumulative metrics + +There should be a way to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. + +Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). The `__type__` label will be extended with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. + +When ingesting a delta metric via the OTLP endpoint, the type will be added as a label. + +A con of this approach is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)” or `__type__=~”delta_.*”` would have to be used. However, this does not seem like a particularly necessary use case to optimise for. + +##### Alternatives +###### New `__temporality__` label + +A new `__temporality__` label could be added instead. + +However, not all metric types should have a temporality (e.g. gauge). Having `delta_` as part of the type label enforces that only specific metric types can have temporality. Otherwise, additional label error checking would need to be done to make sure `__temporality__` is only added to specific types. + +###### Metric naming convention + +Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. + +#### Remote write +Remote write support is a non-goal for the initial implementation to reduce its scope. However, the current design ends up partially supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. However, there is currently no equivalent to StartTimeUnixNano per sample in remote write. + +For the initial implementation, there should be a documented warning that deltas are not _properly_ supported with remote write yet. + +#### Scraping + +No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. + +Federation allows a Prometheus server to scrape selected time series from another Prometheus server. There are problems with exposing a delta metric when federating. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. Therefore, delta metrics will be filtered out from metrics being federated. + + + + +Explain the full overview of the proposed solution. Some guidelines: + +* Make it concise and **simple**; put diagrams; be concrete, avoid using “really”, “amazing” and “great” (: +* How you will test and verify? +* How you will migrate users, without downtime. How we solve incompatibilities? +* What open questions are left? (“Known unknowns”) + +## Alternatives + +The section stating potential alternatives. Highlight the objections reader should have towards your proposal as they read it. Tell them why you still think you should take this path [[ref](https://twitter.com/whereistanya/status/1353853753439490049)] + +1. This is why not solution Z... + +## Action Plan + +The tasks to do in order to migrate to the new idea. + +* [ ] Task one +* [ ] Task two ... From ae2ab9fb1b2811c87abcf4d9c67460239de915d2 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 25 Mar 2025 19:06:33 +0000 Subject: [PATCH 02/34] edits Signed-off-by: Fiona Liao --- ...25-03-25_otel_delta_temporality_support.md | 62 ++++++++----------- 1 file changed, 26 insertions(+), 36 deletions(-) diff --git a/proposals/2025-03-25_otel_delta_temporality_support.md b/proposals/2025-03-25_otel_delta_temporality_support.md index 2a9d872..b787e07 100644 --- a/proposals/2025-03-25_otel_delta_temporality_support.md +++ b/proposals/2025-03-25_otel_delta_temporality_support.md @@ -17,28 +17,19 @@ TODO: add others from delta wg ## Why -Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter-like OTEL metrics (e.g. histograms, sum) can have either [cumulative or delta temporality](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality). +Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter-like OTEL metrics (e.g. histograms, sum) can have either [cumulative or delta temporality](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality). However, Prometheus only supports cumulative metrics, due to its pull-based approach to collecting metrics. -However, Prometheus only supports cumulative metrics, due to its pull-based approach to collecting metrics. +Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167[). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. -Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167[). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. +This is simple - the cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. -Example conversion: -``` -T0: 0 -> T0: 0 -T1: 1 -> T1: 1 -T2: 1 -> T2: 2 -T3: 5 -> T3: 7 -``` - -The benefit of the current solution is that it is simple - the cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. - -Prometheus' goal of becoming the best OTEL metrics backend means we should make sure to support delta metrics properly. We propose to add native support for OTEL delta metrics (i.e. metrics ingested via the OTLP endpoint). Native support means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. +Prometheus' goal of becoming the best OTEL metrics backend means we should support delta metrics properly. We propose to add native support for OTEL delta metrics (i.e. metrics ingested via the OTLP endpoint). Native support means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. ### Delta datapoints -In the [OTEL spec](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality), like cumulative metrics, a datapoint for a delta metric has a `(start,end]` time window. However, the time windows of the previous datapoint and the current datapoints do not overlap. -The `end` timestamp is referred to as `TimeUnixNano` and is mandatory. The `start` timestamp is referred to as `StartTimeUnixNano`. This second timestamp is optional, but recommended for better rate calculations and to detect gaps and overlaps in a stream. Examples where `StartTimeUnixNano` is not added include the countconnector +In the [OTEL spec](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality), like cumulative metrics, a datapoint for a delta metric has a `(start,end]` time window. However, the time windows of delta datapoints do not overlap. + +The `end` timestamp is called `TimeUnixNano` and is mandatory. The `start` timestamp is called `StartTimeUnixNano`. `StartTimeUnixNano` timestamp is optional, but recommended for better rate calculations and to detect gaps and overlaps in a stream. ### Characteristics of delta metrics @@ -52,13 +43,13 @@ Further insights and discussions on delta metrics can be found in [Chronosphere Delta to cumulative conversion requires adding up older delta samples values with the current delta value to get the current cumulative value, so deltas that arrive out of order cannot be added without rewriting newer samples that were already ingested. -As suggested in an [earlier delta doc](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q), a delay could be added to collect all the deltas within a certain time period before converting them to cumulative. This means a longer delay before metrics are queryable, so would not be suitable for long out of order time windows. +As suggested in an [earlier delta doc](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q), a delay could be added to collect all the deltas within a certain time period before converting them to cumulative. This means a longer delay before metrics are queryable. #### No added value to conversion -The benefit of cumulative metrics is that they’re resilient to data loss - if a sample is dropped, the next sample will still include the count from the previous sample. With delta metrics, if a sample is dropped, its data is just lost. Converting from delta to cumulative doesn’t improve resiliency as the data is already lost before it becomes a cumulative metric. +Cumulative metrics are resilient to data loss - if a sample is dropped, the next sample will still include the count from the previous sample. With delta metrics, if a sample is dropped, its data is just lost. Converting from delta to cumulative doesn’t improve resiliency as the data is already lost before it becomes a cumulative metric. -Cumulative metrics are usually converted into deltas during querying (this is part of what rate() and increase() do), so converting deltas to cumulative is wasteful if they’re going to be converted back into deltas on read. +Cumulative metrics are usually converted into deltas during querying (this is part of what `rate()` and `increase()` do), so converting deltas to cumulative is wasteful if they’re going to be converted back into deltas on read. #### Conversion is stateful @@ -67,7 +58,8 @@ Converting from delta to cumulative requires knowing previous values of the same State becomes more complex in distributed cases - if there are multiple OTEL collectors running, or data being replicated to multiple Prometheus instances. #### Values written aren’t the same as the values read -Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. call to get a useful result. However, it could be confusing, especially to newcomers to Prometheus, that when querying just the metric without any functions, the returned value is not the same as the ingested value. + +Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. call to get a useful result. However, it could be confusing when querying just the metric without any functions, the returned value is not the same as the ingested value. #### Does not handle sparse metrics well As mentioned in Background, sparse metrics are more common with delta. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. There is more discussion on this in the Querying deltas section. @@ -76,9 +68,9 @@ As mentioned in Background, sparse metrics are more common with delta. This can Goals and use cases for the solution as proposed in [How](#how): -• Allow OTEL delta metrics to be ingested via the OTLP endpoint and stored as-is -• Support for all OTEL metric types that can have delta temporality (sums, histograms, exponential histograms) -• Queries behave appropriately for delta metrics +* Allow OTEL delta metrics to be ingested via the OTLP endpoint and stored as-is +* Support for all OTEL metric types that can have delta temporality (sums, histograms, exponential histograms) +* Queries behave appropriately for delta metrics ### Audience @@ -94,8 +86,6 @@ These may come in later iterations of delta support, however. ## How ### Ingesting deltas - -#### Storing samples When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. TODO: start time nano injection @@ -104,17 +94,17 @@ For the initial implementation, reuse existing chunk encodings. Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. If a value in a bucket drops, that counts as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Therefore a new counter reset hint/header is required, to indicate the cumulative counter reset behaviour for chunk cutting should not apply. -##### Alternatives +#### Alternatives -###### CreatedTimestamp per sample +##### CreatedTimestamp per sample If `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. The CreatedTimestamp field does not exist yet, but there is currently an effort towards adding it for cumulative counters ([PR](https://github.com/prometheus/prometheus/pull/16046/files)), and can be reused for deltas. Having the timestamp and the start timestamp stored in a sample means that there is the potential to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. -###### Treat as gauge +##### Treat as gauge To avoid introducing a new type, deltas could be represented as gauges instead and the start time ignored. This could be confusing as gauges are usually used for sampled data (for example, in OTEL: "Gauges do not provide an aggregation semantic, instead “last sample value” is used when performing operations like temporal alignment or adjusting resolution.”) rather than data that should be summed/rated over time. -###### Treat as “mini-cumulative” +##### Treat as “mini-cumulative” Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. This requires CT-per-sample to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. @@ -123,7 +113,7 @@ Functions will not take into account delta-specific characteristics. The OTEL SD This also does not work for samples missing StartTimeUnixNano. -###### Convert to rate on ingest +##### Convert to rate on ingest Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixName` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. The difference is that there is no interval information in the metric name (like :rate1m) as there is no guarantee that the interval from sample to sample stays constant. @@ -136,7 +126,7 @@ Users might want to convert back to original values (e.g. to sum the original va This also does not work for samples missing StartTimeUnixNano. -#### Distinguishing between delta and cumulative metrics +### Distinguishing between delta and cumulative metrics There should be a way to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. @@ -146,23 +136,23 @@ When ingesting a delta metric via the OTLP endpoint, the type will be added as a A con of this approach is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)” or `__type__=~”delta_.*”` would have to be used. However, this does not seem like a particularly necessary use case to optimise for. -##### Alternatives -###### New `__temporality__` label +#### Alternatives +##### New `__temporality__` label A new `__temporality__` label could be added instead. However, not all metric types should have a temporality (e.g. gauge). Having `delta_` as part of the type label enforces that only specific metric types can have temporality. Otherwise, additional label error checking would need to be done to make sure `__temporality__` is only added to specific types. -###### Metric naming convention +##### Metric naming convention Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. -#### Remote write +### Remote write Remote write support is a non-goal for the initial implementation to reduce its scope. However, the current design ends up partially supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. However, there is currently no equivalent to StartTimeUnixNano per sample in remote write. For the initial implementation, there should be a documented warning that deltas are not _properly_ supported with remote write yet. -#### Scraping +### Scraping No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. From fe7f3369c3ac35284d8f4bb1254236e439f9a8e4 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 25 Mar 2025 19:11:55 +0000 Subject: [PATCH 03/34] Move alternatives down Signed-off-by: Fiona Liao --- ...25-03-25_otel_delta_temporality_support.md | 77 +++++++++---------- 1 file changed, 35 insertions(+), 42 deletions(-) diff --git a/proposals/2025-03-25_otel_delta_temporality_support.md b/proposals/2025-03-25_otel_delta_temporality_support.md index b787e07..af762ad 100644 --- a/proposals/2025-03-25_otel_delta_temporality_support.md +++ b/proposals/2025-03-25_otel_delta_temporality_support.md @@ -94,17 +94,44 @@ For the initial implementation, reuse existing chunk encodings. Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. If a value in a bucket drops, that counts as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Therefore a new counter reset hint/header is required, to indicate the cumulative counter reset behaviour for chunk cutting should not apply. -#### Alternatives +### Distinguishing between delta and cumulative metrics + +There should be a way to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. + +Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). The `__type__` label will be extended with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. + +When ingesting a delta metric via the OTLP endpoint, the type will be added as a label. + +A con of this approach is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)” or `__type__=~”delta_.*”` would have to be used. However, this does not seem like a particularly necessary use case to optimise for. + +### Remote write + +Remote write support is a non-goal for the initial implementation to reduce its scope. However, the current design ends up partially supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. However, there is currently no equivalent to StartTimeUnixNano per sample in remote write. + +For the initial implementation, there should be a documented warning that deltas are not _properly_ supported with remote write yet. + +### Scraping + +No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. + +Federation allows a Prometheus server to scrape selected time series from another Prometheus server. There are problems with exposing a delta metric when federating. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. Therefore, delta metrics will be filtered out from metrics being federated. + + + + +## Alternatives + +### Ingesting deltas alternatives -##### CreatedTimestamp per sample +#### CreatedTimestamp per sample If `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. The CreatedTimestamp field does not exist yet, but there is currently an effort towards adding it for cumulative counters ([PR](https://github.com/prometheus/prometheus/pull/16046/files)), and can be reused for deltas. Having the timestamp and the start timestamp stored in a sample means that there is the potential to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. -##### Treat as gauge +#### Treat as gauge To avoid introducing a new type, deltas could be represented as gauges instead and the start time ignored. This could be confusing as gauges are usually used for sampled data (for example, in OTEL: "Gauges do not provide an aggregation semantic, instead “last sample value” is used when performing operations like temporal alignment or adjusting resolution.”) rather than data that should be summed/rated over time. -##### Treat as “mini-cumulative” +#### Treat as “mini-cumulative” Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. This requires CT-per-sample to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. @@ -113,7 +140,7 @@ Functions will not take into account delta-specific characteristics. The OTEL SD This also does not work for samples missing StartTimeUnixNano. -##### Convert to rate on ingest +#### Convert to rate on ingest Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixName` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. The difference is that there is no interval information in the metric name (like :rate1m) as there is no guarantee that the interval from sample to sample stays constant. @@ -126,53 +153,19 @@ Users might want to convert back to original values (e.g. to sum the original va This also does not work for samples missing StartTimeUnixNano. -### Distinguishing between delta and cumulative metrics - -There should be a way to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. - -Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). The `__type__` label will be extended with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. - -When ingesting a delta metric via the OTLP endpoint, the type will be added as a label. +### Alternatives to distinguishing between delta and cumulative metrics -A con of this approach is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)” or `__type__=~”delta_.*”` would have to be used. However, this does not seem like a particularly necessary use case to optimise for. - -#### Alternatives -##### New `__temporality__` label +#### New `__temporality__` label A new `__temporality__` label could be added instead. However, not all metric types should have a temporality (e.g. gauge). Having `delta_` as part of the type label enforces that only specific metric types can have temporality. Otherwise, additional label error checking would need to be done to make sure `__temporality__` is only added to specific types. -##### Metric naming convention +#### Metric naming convention Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. -### Remote write -Remote write support is a non-goal for the initial implementation to reduce its scope. However, the current design ends up partially supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. However, there is currently no equivalent to StartTimeUnixNano per sample in remote write. - -For the initial implementation, there should be a documented warning that deltas are not _properly_ supported with remote write yet. - -### Scraping - -No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. - -Federation allows a Prometheus server to scrape selected time series from another Prometheus server. There are problems with exposing a delta metric when federating. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. Therefore, delta metrics will be filtered out from metrics being federated. - - - - -Explain the full overview of the proposed solution. Some guidelines: - -* Make it concise and **simple**; put diagrams; be concrete, avoid using “really”, “amazing” and “great” (: -* How you will test and verify? -* How you will migrate users, without downtime. How we solve incompatibilities? -* What open questions are left? (“Known unknowns”) - -## Alternatives - -The section stating potential alternatives. Highlight the objections reader should have towards your proposal as they read it. Tell them why you still think you should take this path [[ref](https://twitter.com/whereistanya/status/1353853753439490049)] -1. This is why not solution Z... ## Action Plan From 320f204f983a0c4012f2e6631c0c349574587ace Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 25 Mar 2025 19:26:26 +0000 Subject: [PATCH 04/34] Start querying section Signed-off-by: Fiona Liao --- ...25-03-25_otel_delta_temporality_support.md | 37 ++++++++++++++++--- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/proposals/2025-03-25_otel_delta_temporality_support.md b/proposals/2025-03-25_otel_delta_temporality_support.md index af762ad..5632b72 100644 --- a/proposals/2025-03-25_otel_delta_temporality_support.md +++ b/proposals/2025-03-25_otel_delta_temporality_support.md @@ -21,9 +21,11 @@ Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167[). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. -This is simple - the cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. +Tthe cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. -Prometheus' goal of becoming the best OTEL metrics backend means we should support delta metrics properly. We propose to add native support for OTEL delta metrics (i.e. metrics ingested via the OTLP endpoint). Native support means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. +Prometheus' goal of becoming the best OTEL metrics backend means we should support delta metrics properly. + +We propose to add native support for OTEL delta metrics (i.e. metrics ingested via the OTLP endpoint). Native support means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. ### Delta datapoints @@ -89,20 +91,24 @@ These may come in later iterations of delta support, however. When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. TODO: start time nano injection +TODO: move the CT-per-sample here as the eventual goal + +If `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. The CreatedTimestamp field does not exist yet, but there is currently an effort towards adding it for cumulative counters ([PR](https://github.com/prometheus/prometheus/pull/16046/files)), and can be reused for deltas. Having the timestamp and the start timestamp stored in a sample means that there is the potential to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. +### Chunks For the initial implementation, reuse existing chunk encodings. Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. If a value in a bucket drops, that counts as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Therefore a new counter reset hint/header is required, to indicate the cumulative counter reset behaviour for chunk cutting should not apply. ### Distinguishing between delta and cumulative metrics -There should be a way to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. +We need to be able to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). The `__type__` label will be extended with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. When ingesting a delta metric via the OTLP endpoint, the type will be added as a label. -A con of this approach is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)” or `__type__=~”delta_.*”` would have to be used. However, this does not seem like a particularly necessary use case to optimise for. +A downside is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)”` or `__type__=~”delta_.*”` would have to be used. However, this does not seem like a particularly necessary use case to optimise for. ### Remote write @@ -114,10 +120,24 @@ For the initial implementation, there should be a documented warning that deltas No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. -Federation allows a Prometheus server to scrape selected time series from another Prometheus server. There are problems with exposing a delta metric when federating. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. Therefore, delta metrics will be filtered out from metrics being federated. +Federation allows a Prometheus server to scrape selected time series from another Prometheus server. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. Therefore, delta metrics will be filtered out from metrics being federated. +### Querying deltas +Two things to consider: migration and use cases +`rate()` and `sum_over_time()` +from other providers etc. +first stage - delta_rate() behind feature flag and sum_over_time() - quick, can experiment +non-extrapolation is more important -> less regular spacing, harder to guess correctly +next stage - rate() + possible new function for doing the same with cumulative metrics + +should return warning if queried with unexpected type + +### Handling missing StartTimeUnixNano +Keep it for OTEL compatibility +use spacing between intervals when possible +non-extrapolation ## Alternatives @@ -153,7 +173,7 @@ Users might want to convert back to original values (e.g. to sum the original va This also does not work for samples missing StartTimeUnixNano. -### Alternatives to distinguishing between delta and cumulative metrics +### Distinguishing between delta and cumulative metrics alternatives #### New `__temporality__` label @@ -165,7 +185,12 @@ However, not all metric types should have a temporality (e.g. gauge). Having `de Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. +### Querying deltas alternatives + +TODO: these are the top ones, for more see ... +rate() to do sum_over_time() +convert to cumulative on read ## Action Plan From 43f7267dc9dbc1bf68d2cbc33bb21988078d9c1f Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Wed, 26 Mar 2025 19:46:49 +0000 Subject: [PATCH 05/34] Add querying section Signed-off-by: Fiona Liao --- ...25-03-25_otel_delta_temporality_support.md | 78 ++++++++++++++++--- 1 file changed, 67 insertions(+), 11 deletions(-) diff --git a/proposals/2025-03-25_otel_delta_temporality_support.md b/proposals/2025-03-25_otel_delta_temporality_support.md index 5632b72..d931dcb 100644 --- a/proposals/2025-03-25_otel_delta_temporality_support.md +++ b/proposals/2025-03-25_otel_delta_temporality_support.md @@ -64,7 +64,9 @@ State becomes more complex in distributed cases - if there are multiple OTEL col Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. call to get a useful result. However, it could be confusing when querying just the metric without any functions, the returned value is not the same as the ingested value. #### Does not handle sparse metrics well -As mentioned in Background, sparse metrics are more common with delta. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. There is more discussion on this in the Querying deltas section. +As mentioned in Background, sparse metrics are more common with delta. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. + +TODO: would intermittent be a better word to describe this behaviour? ## Goals @@ -104,7 +106,7 @@ Currently the counter reset behaviour for cumulative native histograms is to cut We need to be able to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. -Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). The `__type__` label will be extended with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. +Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). The `__type__` label will be extended with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) When ingesting a delta metric via the OTLP endpoint, the type will be added as a label. @@ -120,21 +122,74 @@ For the initial implementation, there should be a documented warning that deltas No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. -Federation allows a Prometheus server to scrape selected time series from another Prometheus server. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. Therefore, delta metrics will be filtered out from metrics being federated. - +Delta metrics will be filtered out from metrics being federated. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. ### Querying deltas -Two things to consider: migration and use cases -`rate()` and `sum_over_time()` -from other providers etc. +`rate()` and `increase()` will be extended to support delta metrics too. If the `__type__` starts with `delta_`, execute delta-specific logic instead of the current cumulative logic. The delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. + +`irate()` will also be extended to support delta metrics. + +Having functions transparently handle the temporality simplifies the user experience - users do not need to know the temporality of a series for querying, and means queries don't need to be rewriten wehn migrating between cumulative and delta metrics. + +`resets()` does not apply to delta metrics, however, so will return no results plus a warning in this case. + +While the intention is to eventually use `rate()`/`increase()` etc. for both delta and cumulative metrics, initially experimental functions prefixed with `delta_` will be introduced behind a delta-support feature flag. This is to make it clear that these are experimental and the logic could change as we start seeing how they work in real-world scenarios. In the long run, we’d move the logic into `rate()`. + +#### Guessing start and end of series + +The current `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. + +With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. Therefore the delta implementation of `rate()`/`increase()` will not try and guess when a series starts and ends. Instead, it will always assume the series is ongoing for the whole range and always extrapolate to the whole range. + +This could inflate the value of the series, which can be especially problematic during rollouts when old series are replaced by new ones. + +As part of the implementation, experiment with heuristics to try and improve this (e.g. if intervals between samples are regular and there are than X samples, assume the samples are continuously ingested and therefore a gap would mean the series ended). This would make the calculation more complex, however. + +#### `rate()`/`increase()` calculation + +When CT-per-sample is introduced, there will be more information that could be used to more accurately calculate the rate (specifically, the first sample can be taken into account). Therefore the calculation differs depending on whether there is a CT within the sample. + +TODO: write code for these implementations to make it clearer + +*Without CT-per-sample rate()/increase()* -first stage - delta_rate() behind feature flag and sum_over_time() - quick, can experiment -non-extrapolation is more important -> less regular spacing, harder to guess correctly -next stage - rate() + possible new function for doing the same with cumulative metrics +`(sum of second to last samples / (last sample ts - first sample ts))` (multiply by range if `increase()`) -should return warning if queried with unexpected type +We ignore the value of the first sample as we do not know when it started. + +*With CT-per-sample rate()/increase()* + +In this case, we don’t need to guess where the series started. As we have CT-per-sample, if a sample is before the range start, it can't be within in the range at all. We still cannot tell if there is a sample that overlaps with the end range, however. + +1. If the start time of the first sample is outside the range, truncate the sample so we only take into account of the value within the range: + * `first sample value = first sample value * (first sample interval - (range start ts - first sample start ts))` + * `first sample value ts = range start ts` +2. Calculate rate: `(sum of all samples / (last sample ts - range start ts))` + * Multiply by `range end ts - max(range start ts, first sample start ts)` for `increase()`. + +#### Non-extrapolation + +There may be cases where extrapolating to get the rate/increase over the selected range is unwanted for delta metrics. Extrapolation may work worse for deltas since we do not try and guess when series start and end. + +Users may prefer "non-extrapolation" behaviour that just gives them the sum of the sample values within the range. This can be accomplished with `sum_over_time()`. Note that this does not accurately give them the increase within the range. + +As an example: + +* S1: StartTimeUnixNano: T0, TimeUnixNano: T2, Value: 5 +* S2: StartTimeUnixNano: T2, TimeUnixNano: T4, Value: 1 +* S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 + +And `sum_over_time() was executed between T1 and T5. + +As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. + +`sum_over_time()` does not work for cumulative metrics, so a warning should be returned in this case. One downside is that this could make migrating from delta to cumulative metrics harder, since `sum_over_time()` queries would need to be rewritten, and users wanting to use `sum_over_time()` will need to know the temporality of their metrics. + +One possible solution would to have a function that does `sum_over_time()` for deltas and the cumulative equivalent too (this requires subtracting the latest sample before the start of the range with the last sample in the range). This is outside the scope of this design, however. ### Handling missing StartTimeUnixNano + +StartTimeUnixNano is optional in the OTEL spec ... Keep it for OTEL compatibility use spacing between intervals when possible non-extrapolation @@ -144,6 +199,7 @@ non-extrapolation ### Ingesting deltas alternatives #### CreatedTimestamp per sample + If `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. The CreatedTimestamp field does not exist yet, but there is currently an effort towards adding it for cumulative counters ([PR](https://github.com/prometheus/prometheus/pull/16046/files)), and can be reused for deltas. Having the timestamp and the start timestamp stored in a sample means that there is the potential to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. #### Treat as gauge From fc1293331ad2f199bbad77b71b94c3802ac74261 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Thu, 27 Mar 2025 13:43:03 +0000 Subject: [PATCH 06/34] Improve querying Signed-off-by: Fiona Liao --- .../sum_over_time_range.png | Bin 0 -> 33857 bytes ...25-03-25_otel_delta_temporality_support.md | 95 +++++++++++------- 2 files changed, 60 insertions(+), 35 deletions(-) create mode 100644 assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png diff --git a/assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png b/assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png new file mode 100644 index 0000000000000000000000000000000000000000..3d0367b588d7b753e9568095384043353a0c2e5a GIT binary patch literal 33857 zcmcG#bzB_Fvp$TwJBz!!JM7{fAOs68OK=MkToy=z1SjYQ4-$eC+!J&Igy0$^2@b*K z9nLxT-tT+w`S-VF(+3s_v@isfpLuRl~;t;UFO);cKWX8z3Q}fsl}pMX@k| zGm&8ZP9!9pkIoQ?z6J!ssPFCJ=B3dBJnCU@fh3pwNLarSES^=@~0dU z=8^*vkBo`n*zay!U#2=;tii@;>}NQt*1gz1n_vd1XckQK#M_D*SaNkFl(kE3G3 zM<@$(lj^jBf-opIr#NZe$%E6%)V;|KHWY@N>i&446?`#Q6=%BT5EwI2$H}5y=lJ#O8uM1!lc#7iEZ-U#Iayu#YE%uGWDAF^7RVZ zq;P6Yvq+Of#apaXnmJna#jTT-#g!#F#U2`6ZEt9zU1l* zIkp838Fsbw`ZU}Ot;_{RfA#&Bi@1U9?VatJvQIkILqFUEUMg{ezrlm_rKTIFL#AI% zGoj~3Mxbahslbia;=q}`Hx;>Ex_-Jv`crPo$Cj1c z<;0uQykZN$*&(_v95ve>8^jAMU1 zrIPx=v?fsZgzf2@@#3@~t1gX~r<~`V zN09gEsm_w(Qt(pH(shus9L%zA`FV$K2X+YRE%es!YV1k?FAdKGBM{>c1|dEdtOk}1 zYlQi~_Eg0yGAhbbl}NC$GvBb^c(y^qUK^i>y<%Uc(abe7-D6~J<}&Z3DsK07GwWLp ze%90|>L~75&e+Q6_dEuPu#ubb%N1T%zDeFQQ`y?I=F@uK2G_=5CkNMokMEm4+39>d zYqE4$vb~)d?1wqkk8IWOeG^AK&vT32dm{5f=FQWnrMQlokoJ&-5C%vDaqnz*%Wk-z zr@zUj=loh~QJ!7XhR~2g-e|5t9)VPciFIvwv-L`(l9QWL(+T}{DCH-u_K^wx+ZwYP z*Y9TE%iN{iEt+rL`?OgcaZUtJ653+gKHnB!N5>WoX(oy5^-m4e_t%eTE%QuP4*&kx z+~l9)-*H}^R6T)HAZ}Ilq3eKRMLt9NRo{ENgsZvh>eIGWy)(k|56593Yg$GeD1J2R zElM+nJl-TO37QyYl(q5U%&Hm_22(auSIjyX0s{vTu&biqMn6i#bduR7|DIqz zPPZm?G91tsKOXN8FOZU(+Qgo~+Q$BbwQK9eVdaH~VX)!zGVNlFGJ5sfe2To8EM4^^ zJq5O8%3>UTd#F9{$PUF@tXDC&*@@W;!fGma?`}=?XV$AST+tgeKJkl_-80gE_WDvW zewjs+{vtSgQMjk>_eetK2P^s9?Y#35oKc+dmx;mEH)5BlJlQ zeqeH69c}L2IQ_2S-RTI%IIL2^^v>vV>O3KyrcHd#ZN8DZ*|fIR|JCn|CB#APGQR$<%SR>Lt?$d`J3f6>t#%9<>(|{M zH*Z^fF1BmD^7Pa7$;3^;!T!Jr&r^l z3Yq@yRXVe0pX4e+YVDkdqE3Xi4=k`c5Po0p*EhA6GspdfWhS0J?+m_rxsh)8g!)O& zhq+-X%y};{9x|uu|IVHKJjyE0t z*A##LtZXiacnRz+FI-EVb1+ne#)X(1tDij@uxlJ@)O(#TdbaSVfclWO4xZQ5u z@2#Dm9<0Q;62JIFu82V9U`ILJM?y+7K`Mt@>6%GSnd{+-o+t?FtSTI7-^=kmM-pv8 zCR)S@-+qSlOTRklnXq)J**h!=By)4EL01OZrFy-1jH`)c8kE&HdwR>+5n;pH3ip}q z_Z1PWzgAAWknq3ZiaFapJq3K&mkuTxjygI>+`usw5*jiH2^}~>1~z$Qn*TmlMdn08 z{p&mm5>l))652oa=mPtPPcpDQl=*v)ngT<@1pbi#TRx`r zdk+ul*yKw0&l;=n&SW~i^Pr?h~8zrR1fzX-pFx08U7l$4Z! zps;|jFduLSpHHB>uT21-yAR9XmHbyd$__qu-p-!B&K~ZJ5B1vEdieRuF*83j^xvPq zpVJ}0`F~n+_xYz=KnDdLt_TS63kv*q-9S;7XPE9$^VoT5)>2s&!Ycv>3$zHH)TL^8Fe3hTc;3edHQe|z@0|7Tt6N9(*j zGe7Sq)w|duH#X@RlR0QV^4NQtEoc+z0);|7M9KZ{4E5wTJMXSsBz{-bfBI%@Uj5_r z_GpN~y+4k4KS5&IvmoRwJLmbl^AgCV>X9)mo0T@EF)#A)z>@#U-e3~L!B+kY_wQ}~ z2cwf_l@9yW&CAD|rCOQ%HF^c|4o5*p1C&dZA3A?6zc`SS7$7rjiJ`Qd-D{ZAbr^t0 zxg;s(ZxWd`no#sz^-BM)+HlOWfu%6urT3*g`l(1vgiKui3>c&LS7#167nxR}b9pcJ zTjeL8KA#(K`}DIk+roOb)@HwH!QJPHg_jEZ(PINvHTuBf>zk8_5SF8=`f+Y4XQ%wY z(igXXzT+mYgkJA08JoL3uGOPy)LETSxc6;4=xm$#VRd&*=|3toWS_F;zZ^IORPV4~ z+ehGYGNBO0?q#gWDzW67`eFwj8|)3kqc-raqYpk*k_b3ljohF#wI}vpkLRG+ol?Ce z{#;2LdcNep|Gu)uTZ{K&^n&dxzGpbQJRjQE$ECJu3(D-0TTAtYD=r5 zDJ!Bqiwuv3y=_UDvu7rRN5Hyo&sPq1LbQ#K>oxqg2PY}u5%hDwAc^TpCgkPIy$E_^ z1COD2dG1njhW+Dfw^BFwGxK3CUu`#v7pP(H>S7px^efCqd_>XAWM74u+1yAIEBtvcn zHF>JZK79YyS2{|;maHaq^XI!2-GP5(G%SASF#(4+!s@Sar3A+H*QRb}vK}@*o1!M| zB*iNj91{D<+@s^$kT0M$ZP|`*xri&pf5#}&e|A^{;C=Cq2@LI%&eh|XbjMh-B6+*FaUKq&)!!o;tI5{QlVJi zVN;m93N>H-svIIuH2+3dRC06p1Fx?Z`z z+m^pQHIx|Pt-tWvtgNcjd>>T-sN+yhQO&1c@8$Q^7(%9pROUVFJ~zHN-FY-Ught*S zc7MHcM>{w!?AwDP$1l)vnsWUkRqrNjJ;(jiPuk1A{-9y5vIfX)9W8vVC`sPCNHyVQ z-dAGeP(e@4*p0Ex40Ha(>mi1)(+oz2&uWvl^;;9s@Q9YWOGQYQ=5kjRb*wQN(?UoPr5tup(OT3sGkXM6ECl}PM139 z_zOpf=m>vX)dbJ?SC`?jNgN*#E<3%JyFVTg zShBjmdZH%9HzZ}5i&gE56Hzt7WXmu?y!l3v+A`>8$r7#4=$f&G*FdDK@bpiG`^zMS zd=K6U+7+7Rg)ayLsUNg88mrU=9-WsdW!G-BLI&ft!i(M`d>bC(1U#4WM`LULxwQVj zXU4*EJPk$!iv-g|eBb4Ie-=|`(Y|mbvF`Pk9y4{<(k_PpS|KY>a7&E*pjf=4CL#NR+>V3MeC}r0*^^a_IUJr*vib+(!}R zWlsuTS_iT&>1Eo!ck>F8qN`ecwWT&Z3prN6od8bsM?QVf7d_q6J ze<<+qgblfL2tO#q<+j58PZ|PIGvWy%q7>w>N%=vQ+8eQqd#!Uev2?*jktf8MURRnd zQhcuMM}wWOb;RfFk0!QUVg^Mv8D%NqyXPxm=R*`e%p+oZms@j7J`-}cN1~r=+K)8_ zhz2ROs#rI8)u4(k#iOO@{htBFSiV1(sT&9SS$_RU*Zc;K7zDICG~d>Qn#Moko!h7~(CTd%9ful-dEklE#J70>o zd${Lv+?ujIii7WO&-W8$_76pPYIRL~#>8$(KH6UWEIaSYo{|nNWQGXfb+nj(O>$%D zgP=-!FtPU`_B{-sHI7`*NSU9-sW+BEJYz*+o>|vJ3U_NSc%22mB39cGi@CSjVj2@X zkvMXhCR9XF=-Su{SPG1af+bga$~<+*jEF1lfs~}U2tRc+5sJjN#U^jmH)67~@o?4B zuIV_NkHknRc{(!Ap|6QLWbnl&UJN;c3smhjabhEqzX;gYe%Jl&2gnW!Vs$lppbq&r zYE>*zewFJZuc7$}3>=4X5H}v5bzB5`<>BEb+wZviEsQsFA69}-#+Gp+ z6M#iyVNC41MW;4hz-q+NN37Viu&TJ7%|=h2?K2N?J?-kbCN#8k{7mgRr( zJY-}nKa^h$QHLgAGgtoYT`F!9B_j}0i#*5gOK|Y9FeF>wMd)^0q%J=}>BtdvsLCp%DCuF`zeP(^$Y0ey(owMo)Z@HQS`zaW1~BEhOPh=(A1ZM1hZMi}^%ytyPVgO7wl?uyY@3qKyT!Bw%O_7o=7SD<7i?p!^SL+q%^tCO1bgCiBCT33$ z@bfkDqR(<#Q1=)Nk;elDq0LU6h>2_l5F2l&r)$8h$+|yCVkR}`bS0b(8&8MI{_2qk zQ?oyY*I}pPs&aexC!30r4BKXKI??AMS_9QrQc=q7r609dTeqoq=Nl_mO4|h?sXA*f z|0J2##yGH7(;n|0@iRtmU($VLP{t)CxDY>aD=7;1)>9|lj!zX8M;Dnsc&?>d(O^A>ICv0{M{Lv;ZmFeyo)&M@%_>BYCWu^3r$dGv*Tr)F2`3 ztaQ71*9eTS9p}rxC90WJlGqYhK%}SKdo3G@&jZ@~3H$xNKizJ2)<=A1w&xO!u78A> z`gFOKbL|ox*1JXt=Qx?kiuQtbcw?o_cVoxWnErB(o}%JlJhl0#2t0;$buHj;UfL1; z;jI4>uBZmd4JSuS4*{EGuQ=wo(cbrUn>+)pq2-(BTjAP3*mhM0u24qMK{9C-_Jv1! zsIQ4EMYP~dr;!scdTCW*&OKGMk0Qd80|8g_>W@AYYX$pG?Y4sZ!J*5DhIQTUc`Tn3 ztQhuh6%*3+W5AXfS#9^i%eA4^w3CcZP_#=M$;VKk&;ITQ9p`7@=bTilZP9DVSWDXl zVWGT^FbSfsz)H}mjW&lj>_bL$2%0lAC9;E=W(y*8h_DfXs+8CnsV=eItTspO6nESlNUj2TZG{#pueo z+$VT;l7Lrn%Vgsqzx$1FqIrHIDI@A=rS1R7;W<`Ch*h)bMJZpQIKOt-H1m zLXxQDA8AS{2M-a8tW9bOBvhaD+5|dND1Z5VLN~z0O+jd~X5&oRNm@q+s&y?EiAZ*Q zY~zd@Ykg#CV`9Rky&KH;Z&Ka*!CQqCoxX1ihRl}pqvLeq5U~R4JJ_Z4Q4=I5b(%4G zqEwWZ&l1d%$qBY~#K;)mBXs{NS_RE2? z+dQ?xGz?v#SV!5F8N2vC$lznrm)Q8C^AP01MvK_K!roz72MM<=Qpb0za*Ew%D34NZ zzS3y2#GJCxiPRO$2@o;YzO02CeQDTa6GddzQ$xprfYMRK!bBRJdP3vT{xzpH2SN0b z@OgX2HxCoNEWZPO~mKHa|LS7ZM-k*Py3pM1mH}*2Ju#} z8oyi0WRrE#b?S|3_*Q5euuSVd_@YD+7TD_0&*UC{D$8%c5EHL5e>N%AMmFyf(f1W( zOmTuWzF*ifPYJ4Acb`qZkj=out0=6~D71}fM02b*!B1{NNPs2j59PPBj(e^wIuFsT)5uO0gxw22^Y!g0NV=2EA?pT28D<212junIf}?9Q<{p1C)y?*!b)RrGV_n1rk2*qH`;U=%P^n@< z835Ca2xDE=h~8^j7Cf4ItGJ|s6<#}u*B_ex!gR+rNHZ(1)L=Yh8UgK zW=t(CPnNq_jS4lB;g&xcm-(PjJgw>nr5Gc^)$YXp@;hdRaDZ1FZ1ZE*F6qxYx|ahm z;ghp-#~PicFu4ZD+;M0+q`cN!$kSp>0_+FQBwFFvgpF3&py)u)z;mJVagA!Ne$GJe z*#&_OXR;|d+XqVV-de0~p~}FeX3?15IGvmk4^td`_|R=Jd1eW!72l|81eBT}H5p+Y z{XQ&PO$x8z2;n8rpv~)OOqqY%peW_YNe5xinxXhuGwn@fNwZxAgoelCiBmOBosVm+ z*JLUk!ibx6{LzDw05OZnL(G!luVTgL z%F4>KG?UCXLqw>P3}L{B+i@p;^n$}LIU%IbeF%Hq1(0 z&?J)y#O=BIlDJM!*UUt-Oef0$*8y;$1(1l&4h5QKZ9}t&FaeDk!pjj>HvK3(beVSG zIjl7pA%Yf9bY*oU^G>1o&CjPt3yN4{9ra`&s{IHtQVM<1ACv7%u^oK?Se59c{+3l# z5iSm<-}D^K66)kt88H}P^2&~(by$L-m4>WnpQ>Mr zljw2I#)4Hcz^LDCpyXg_-S&Md?X$0-)f?ev+?4T=6Y*(J=HCeu4?!6mb`CD;=LjjA1&`j z_fTIU>d#$FT*(a5bcL~OrY9t8rP>D#jfkIwI;PH$#7 zvHARmD@J3%NZ+yZGU{}&Xlz_XUZdEceta{CfzQK(dE+k6U_!XT@w6lw08|+BKTQTV z%?A3fgfzEpB+A^dzuP8Oqg{jQdwCF9?t(;5cS5$PBBL>q;Ouul9uf4-v0c<0@;klP z1R)-fLAxzXlobisW%b8FP-xEa$caa%7WfSiN>kx{mtWsWx}^N*ma#Pv;n&IXfE!RQ zwa910V!2U_Zlf-k;^M>W9fv^&?7Z&-IFczl`xZICoa1SI{RLGDf8(ok4>)WryygaX zlS%Y1oB#j^1U|PvKK0U9v4c5iG5u7o$yF7LRFRBAO(%EZoE58R*Pp91r#H{5fNU&+kseX?4#$Jgs|+S8R~K!x;!Ev9ZAi|(|fS$or^JVD83&^!70}Z z_F^MJVz+GCVYX#0+?)HvD)=bIqsqKr^j&SVCWdx_+q_n2k(W~KAQ}Zm6-#imTm{e- zte04Hj&f_HZ!RG7L}%);=-BwG@2)Zf(;(%m;X*f^G5CL89CUnNdcNSs*W2fWy>Od?h6) zz%n|-Af(^jzHQ1kl1a`lkUN|0Ggq3G&&bF?Ew{CaQR*P3`RYiR~v`2nYd%|!Q%T59Af3Uoq zqd?vznAXQCtM@xk%X)YAV~xj3r#!KiKt}^Zh8 zU9-YqRoY*zr2uGn*J()9E6tFGz+S<6v-_lT#>{1ozpGeDxi8LoO6DqoIm{JT1kXV{14jTW+JG48VT6Y_oi`mR}8Pcj58BJ zLZ9&Z&K}g-Vc_$M^oBFWl_+yYMt8Y%1Y!MnOrUUc2vN}P1zi(i#}e5#MiEV%bKs>R z_se&tW4Tid^TondY@l%aqq#|5f9)Ujw@P+^u?E0ef4qG1h6b72>~t5QI9L=s$$&Z} zWRH?+P?};ddp>Qb=T_!F>wUWObs71Ion4Xtpt_k158|SyH{!lHYK_(zziC{vpvHU- zZKt9P8REg&Nu_;*YdDsit0#Zij~kgL#eL%JAR^20+iw|Pb|Xy=7O;9#Q#3`i^)r?s z^icbg|H1y$RQ5r*;8U{?mMB&sf6~KB12e#D-QLkVgUL+6u)~`Zh3fYCqd;C_bI#b{ z*gz}BLO)+nZ}5?&VwU1fxY{I%kgW0Vzb!Rg1F%$wWy%uGhalm9ecz`5F~j?IH_e8B z;Zuy*fp{)G&?=4luPOI04gY)T0)ERi0abs0+{E?uUkDfdY(R=eTOp6$|F8BQ{uXir zZb13)LaFkr|I4~4I*=qI^s~=ZDoFnS?L9Oc@t&0t4(ImBApubF|3fMLKklfX>$_M2 zx?i2w`!dY{cGwE2wCvO?y)3~VF;P)_kW;nF8Q>+gN-PKT`Td+6`15nBY^JX4MwB=J z040_H5c)^w#X7Fl_*SluZu4j6S1mtFb87miZZEeB5|4FTE!E?%C zIs0F3W?r`lwC&c7%I{-QK6&bwZf+aTwg^D-Z?xuJ*d_sN{EOTv0wOO1f#Fr8yK*t; zpvhnFiueO9Nk+pOYX8ibzldn&{}&Ck zwCFRjHx5usX2WJrw>ANg**nw1JN|k#itz0w1M%v`Mv}jwj@aY+mHD;}0O|HsxVtF2 zr=SN|hP{Lu$BD4b^e2l_+qoMj{<|NCKCxw5)H$BKfBnEgDpmssl0J{WaBOfi!}E6& znouec$JYx^1tE(UM;lAPf_%_4|E!YMz)I$k=3o6E{UXqHvwnBh+|{}gcE5Bl3h-e8 zJSb;&OV&0^e&79o<)PVat0@hkFKvd6UsC`6l!YrSmhO4h#wCzNIwtDgdAa3N5PDV7 zyk*Cbp;A^5`Y9*$>gU`juR4?y-@%084MHCzLdW)?w0Sy!IXbOG;IzwcO`YZfK-@jQ z`cq?Y--oWdUuR4HRt(M2FqHlZ8WNDf$LW6EApL?D`%(p&_!5hVy-&Yu#(61s_!Iyr zllC3z8Za06;qumoTTT2Ve zbVHwtJ~UkKx5-4y4PF+6-CcTZ3{FOP5ClEKUT|-7+n1MS6{WhI$s_@pIT_)rttO%2 z%2a`B&8*9*Y-j*ey;7fqYVphzgT~@ z4}^Uu8!w#RIs3(xtThv?vwhQDlZ!6rrE^FGi49w&dM9j*mSmU>XX@}r zZjxE3lYU>5iy)Os#Elxz$Y!#?uG$DASQxYFN#o z82sWfO??sCTleC8KlJmkDvcZ2kn>M273gZtm_+))>g-3mkiZQZJrMcdE0uKq1Bm8i z`SpoCtt7*aThrX&*NYzPwK0j6-6;!gXyLl|R~1U~vG0R49=l^YLf8rbPZAQt94MHM zoa@#|0R-S=xC~$HD$pj?apbxUd)k1QxSiwn$5E3G%T9Sw;xWGXg8vYme+xiyFFbi( zUYs>++f`poqVo1o>A}JQBqic8=vmr+Wn?L*b#Z>#ZW1EXxm{g?k$zgbu}A2 zLUjo+lixfao(H$!&Bb0fE?=Srh&#Vt_X?eNF4M_-iXyak5O&|D$D4lqK*@i&KPk9B zO?U1Wi$+1KwJ&^^J9|KzWNj15X*kB3`Wdd_0CmId0Kco7%dIXbfd&%Fr1zfKART4e()Czk;B@F>)LO|JhScmc~McUq`wswc6TZN+#vF9u+mmmN?s2~Hzp=ot9pF_mJ^ ztN!5`hJch1gjZGNx~7pe)YXKLRZf%-8{b6WfxwXW$b}pzq}M!fA?HDe1-(_Opa3e& z5sy=5k{u;@kFE!E$0Z*+x8`eS4nzPPv()8#O{Wc6&@O|Zwc;WacY-Ozgc`B&Dqd19 zbB2g%42z3Go2=F?<20H%1F_&7p-C}AZ1j$ZHBN$z?tzzF85uru_7bMgTILIi*hyeQ z2=-g#V_?Ix5Jbk|Qi;ox4jHC{Nk7Xtz73vD?evP8z^|~PT)$ z9p_H5s4W#RHhdFff^~&I|NG(f7X-0S0QF0h=lmsv9g3o}Xvte^5_%pK-G{P5=lz^oS$Gpha70In(spd__8g>aV$&8RUTiNm+!3VFN^Ffss{pfe z==1)Bh8(ossQ7`bLmsb)F*>2VmQwn4s_g2b!0u)T5dSa9oiA=|SPxGCDH2~lP?Can zdZ^`tTNE%jB8|3q5eU{X;|O~*L0B)R5n;rk)Ou@4ifbZs-=I)>Xubl?9pth%vRhd+ zh_u3uCMy)@oTgnLAoo|z0o%VTG}cL`%KmVYjq!&Yw2 zyYBxmQ?6&|eEA%j&(OA#rU&H17!&o+y7M(`=FpU;`lIGy+hBFcT_h+Waj>dHJcPJz zwLKVj5-=MeIaGl{>*Hk)#8JN+T&OO!cU20epm zGI)4W)|?n}mEs3}<}nySmle_aaJ!bs$IupfxkezpsEm@N9N{GwZclW75YiLsn$TvPqHz|Y>;aA%udv(XlsE;DL zlW|Y7->{2Dv+oV$`cI}N=xeHY$1E(xa_3t1e_6e+`1n3wKje0%fp`9 zP)-~~Q3~z0E=w|?xNx=^mtX$MVcKzpUJf&aHo5S?_j!HZ`m0gB>pFn>g*4%?bm+~D zkPQwgDOOg$+-)V`d;#PoK8x@ojW{jHfHccs28*Y6o_H~vg>x%vk}F=iypEH} zj);C|`4$nLEq9NN_1ck%d@x2GI`b3=>(V_6V*rTb1l!s>=`rq|_71)T1E=UTaqHMR z8yN`9Lp>59XAbvt?V6jU70l!o1&qn%J5Hu(H*wKlhzK5v7B=`o8}<%qDu}r$n(2l; zc12HS5Dirssh9X4O1;=rGFAa>}#60Yx#bN~UEI)KqUImw6VANOqT_{Q~>Ykgdu zZ8c}DHAz9P0hxK<=Y{$5s`H9gWbzdyciB%Rh;Z;1uPr-*$XUnphYn@*Jrx_2vWxN3 zxkWugTURW`m7vP~{d!n^Ah*vLT`|BqoGk8(_M3y0h-#3`jGBZalIuI(hz_S{RveLK z!5@9?k8oONFa+9|`MPLU9D+EUD*hXuNLBzKluMDCUp&$HE4E-%>f+-_4G!|GhVY-v zeUw;kl$YYtbnUhVF()!bvfCe-8m5o{s$@$;YQmsU;~DH z#0b^`IHjL;7p~LrLL+sun;1#6$%vM6lUHO&;+|+oYg2M z>d9!B=dMU6?n@~vvC43R7>t?(Q{nnHYmFX6oPGBs_6R*G@`BleD8+UsL5IL9`Gp$2 zSJq^L_gb4(n-moA9g)l}7TxKhDW^8%CyRVTtnY{gkNTz$E$kUt`*1qQzY z!f_bUBLmOX*Iwl0AfV&<3*ONO0m3WsdcRPlNe;pz0v2`D*9-l>+XM5q{43)c)u6S% zYl|nNom^^EScW4Lu8W7rrnI$C$)!d=UD9nEz25BS&49^9;NT7sfnUL@VldVP96P_9 zT4RZ$ZKMNp;MO0}04n)+`gv$X6Xl*cQ_3J7*;jDV9{X`K7tb<4S*-}U_l;Ilk>4Qk z-~j1B5Gq1j7KfBNjfrA(AJ6%G?>V#11s2iS7k!g7{LaO11VAc_JYc}g&VohAt-ad= zsLEz|qL5ZF^9^BBM|aJp1Su8uYo4N`W60iGVASmm7EC2tIKE4JgLR+xFgb{~#-Ov+ z*j*CSi)8y%ar^;e9|6RLgQtv^#*E@Z%iUTiS+6qVGME0~k874#c{=qcn(BYg>hSG~s zqi&NUPu1ifUi161C;@iOT#vkrRg-eYNFMP%p7G=ydKb|$I$1hv6qNS(gh3xj1!;xo z7h4E7y4Cp$`*uSUe3j#xGDc>Om+0Zh5tv@xj%R{r?S@(O8Job;MJ?e9=;~pJ^MTB( zp$tL03|dgbDeH;6;f^edcJrwJOQN%_Sg!Tu6IbG1CUHPd#nWI;xIOcZklL1bN*lu8 zu?8_P@2*yDDImbQzS3HMO#+j)6uvY;Mv^UW-QTu(HK?aqbH$nD=xlUh3)O|2^o6)mRh@M zkHMfd6&^F8I+R0ZLCK)pn&sAkV$mqzz3NsLn6u*)(bZUC53CTZP6`4Wy2Biq%2`7Uqt6-LRB0Gm?|i8u4i~^t6kd$Z@sTTJ z`KVw?K35lDN)b|vlFW_FY5{PSy8uqz3NQqi7LR+Sq8pfKgSE1@=3sI#k;>>z8Nyc> zF#2OY!U4HuYOHs~JPq_*fcK9eJ&r{+KBmGrj!_|rkcN1Rzdbam{u}SL*8&i*CRrE$ zCIC!8lp%Gno$9Zy|5neZ|w=*cO?WU5Oj`Aa3dAq~T9#o|K$ zzc6xwGO9$yT2;SUpJBgp*HsH*E!-*$PfK!XfV+D;Vh&}@8{NkLu)oySCb1Gc^~o2g5pDED40@)9nM0Y=vMFrqxu__y zVax~&*U41ftqg`WxoRK*XngYmRkbUfu$e2Az zTvqh}Wk{%%c@3I)M-F`ddzRo4P1e=^m`#w;eiRyo8*v<`ZjIvf7SZjs_TuF^=>a(! z9Kfk&WJ((0kJ5a2-#>7bx`sQ-C~yaw^Vj}d#tvyZj>JN5kl)ZS_4aoI^XKru0~r{S zl`L09>G@hy(H@WPG=p0-?Rw13)%EqQmgm!9A_scp*G#0M?U(dsSk=#uDL^gk5J@7e z)r5u7y#=mp?x+kVX#j;VDPF+HSb4cp3jH=T$dru0wn1eE~h;XPH8 z1>%Q!1zNUl+rcydjka8tf?-e){NRm-pu1XmYdAz4Rm}))_;XXtWHq@3d3e?hdeM5M znA#wDF2!)tDQvS!SIL=1nK_*R_8bC%A z!YryS3(@VTnn?T8b1{RW_t?Hyao%gKvliLjU|tXNfa?}2&`EC6yT^$CX3@` zMMbQ#uYByHhnA38+!YTSizfH_h}ou$JhgDAl!1j2P8q6w`W}JE@Zg_`W54j?BOT$@`H}9|S+YhHok#>yCuF)D zS8UjwDx2tMDC$~!!OoSt0zik$)Na9^!#I%`5Yp5>RRe-={D0HbuK`r`GnRd|D(TzM zQk@g^&Y#&1y4ST5(*f2Rii9>MQX9>LMfs1l>HAc}gfQbND7AtFcHo65X?RAgf(LkI zR1c};ivUeUJQ3x-t^f5MQzJk%Us{s(V^nc}W?2K$u`}C%>HIe6t)T=j!Qmv`+vitN zId`kk#{dj{97#x$3=1-mktrqN3dPk?rdyf{=5^B2C~Q zKwo@^arL1TT6Kk@uTX?Km6C&6(Rj~ZirG?Is@o}$2}4a-x(%NbVO*+ChV!!lcEILw zCJQ@C#i;Vnn#?#|kO*Gwj<7Vs4QVRI*Rr06<}6FO1x<_ybS=sO`Dm82f+L0c?W=Ts z+A@wkiu3BEeTFIH&psyGw@~)LY_1YY6-*pF<(Bzu80Gt))2=d?bN-X_NDJ5o0FG~c zP}ai*BeD0MTC9Iya6?x({g*-K?d zh|X~i>rE6bv}L&LP<6K|0h}XY{7FRo zISx2_t;h{}u$yv}2Xb+hoh!ak30N%L7S%T2H9hbX)z=k^M&f|T(?i12q#thm{J>!H zI&qL!G*#kqW0W+FVRnLz`Hkorvy%_LuN2it zy63nFXt0AZW#*|J)V3afe;_I&f*$;Y1AjDeqGsWOqQr9&vm1$H#8)^%qG;6%LedO4b0->}&4& zQN@o{TyqoP6ZgkuPe$Ksd?4XQF(s7*pz9@oGD-7mXdRHS|4if$FqPXBUCa~_n(TWs+dcMW)|W#o%@2C!5AbouI*|Z- zDE#ogj*(7Kx<2vex82VwtG5Vb54)?1#`rn`@@C~0;7M8sKE$aMPGaj@ay9=I`s;rI z2Ejhq|KTkI7$HNxSJg{hwD+%&C{O^NLGLlESwhD979CCdM6}XuN6-V`_#+@@_ubuL zAK}u+A^M!B(JwX7-X|+I#z#GlGtr;`)FK=Z2_7Tt5pB>8{DDb5Ys)Zu#^rzdRV`M* zaT>aHeXlI(F@8-P1)*yjUp8o4L_f9-vBP}JZ1uOcESs|YI6B_J(`G)sy~ zOLs_1NjHc=N_UrlbV;WoA>BwfOLr}K&l}_8?{`0Q=l*r)&b{9mW@l%2d7pmHbDrn* zdL7PruTsEZC>GV~_$IK?bw`5G_*UcQ5BV}7sq<0{A>TYOl~DUBZi zL6T|H9FVTWwg;$|(Z%!xVgqw1tHHgJFI1(TK?jg!mJkp)mZyE0IHgO}52p+ek4B-G z7Geu<2d#bS3!g$i^uqLl-2ucUD!xYdWPs zUlj(NJ3>F}YyB+qM}W;|3HTq;QnmoZIR+BXI?7!#s4t`W4^~2+JCjY}JQIMH1Z^19 z1?!k$ByiL_hkL8bY{P~WBYohP;dM~<6{KZrIo$eLOuRhqz!ITtt{?Q>4ih1!!U%iz$^&c@U_TnvI#rirxIA zi~_tqTlc$Vn~emO0DqK|yd86k6<6K*x!qR`YB}rZFpiteK6M~1IKKwq@ElyuY+AgZ z&QE(`qQEYC*v*4(vja0zig3oFi3ky{^uqgm%C@2&?k*!l0E4awf?HRDeXeS{sFl@0 zJmLx9m)^&^dLnsp(;c63qx%U-IY^jNb&P?-*J9*Sad15#)~N+Hf~l5U$1YBpu5N_m zzK4C6%lcC#NNdg*Ugw;}D(w-aYZy_1$XD*76j1xQg9+Cq%ohPwC1UFKtuS;s#*6g! zHGpiCPzR6_qv`0K) zAA?2_`O#hLpSB+8$w~#bkl*sMeutfztdCQb*Toe<>@^w%!BgtJ8Np z&U`he@1pywr5r}}id${-(iubR0Hk#U*m2HV`XGjH2QtPAFJ;0QZ#IY4fd#SamUs;z)Q3^?tdUF5~~Dk9yi7ZLe9dvB{PsAdZlVsuE(v@zX*jQ z14*EU7V?56Jqp;N*i9U-Ci&R`_^k0~=hVfH1JsLTYy&Xq;~>LMF`oPFwzyPN0Lgj? z+n}0NImE2R4#{Bk=ATE!CmJ=|HuF$c|o>=u+c!d!2L&fOYLtHm6RS-BN05|8@jes>ATqpX4S z@P{IM0J|vziI2vSu(CIc8SfqDqnhHt%da9J7ywx03>`Bl*XX6S7_iQy1?T|k!%+Zl z4DgXh0j*6*fJRP6n3NC2TK7xv!ov#}Zbd^{0NqJ?7lSSq2>fkG+n}ttD}d~9Gy<>& zkH(-6dLNhjOXpJ|ZCxLlv@Yw<#ixO^ZkJIrSCLJ5-vIvT79vS8Pvbgu?oZp<$p^2S zy^>A5Rt08uKXfMF9p_F}`xp0x)*p0%PbH;j;J<;yn81MbJFSOW{u@Y)1Nd>YYvK0= zeukO<%d2!XSWI&3)v_%=LoJ}Q|M>$JAGmvEs>!JTA}heZ3&PdX)%%h(xc>qY^8xlK zY1+&i``^g^|4%g3T>;}5nQk@HxalU?+f$MN(FvPX9$M)Fxw=Yq z?sdqQ)&yg)v_DRV_`CNXJ=OBI!5@p!lx$`&sLbh}G^GTWqk0Sx*QYel$&&D+_qJG9LoQR2XS9o zodUBODy209!yltdkjdY|;LaJI^iaz=d}A|nGT>@g0)h+gqnJ`jTM5=zjIdTTO|*b=PO)KYQ{14O zZJczR$E4)1dKq-oCuFy8;%pj*YZ!X@wdwC ze2U=c*;UFZPHS$++dXj5I}zY?XCE0VcGHjo>bil6+-}}O5m%-G8(i~O0pm1DNq(I% zk~kNoCg(>QE7YYrK!a@U?bv5N0JV>1Z+>D_5Tfc>9&8}8$baX-vlcgn3 zmB-4|s9Sr=CCal!D{v%zR!I7@pT~yMf5+|=D)H%}fueUw zSM)XZGU_#e@?#T^BSpFBK0=YC?HT*tgUQpzuBeSXiL5SH{$}$B{^5P*qUIE@j^R?@ zqb6-^{6gK#vL)X-71c)-)Dxx}Bpp6)oWu1V%WQh_uVO>xZO0Leo#e5)75KZQcpBd? zYVXdMyT|&0p$eJB_+wH8GptMc>0{MZvW*dQoZ~)!@f*0OO-09{sHnvg}ro0dmWLRm2aeV63TNJ6G1j@ zt`N>ZloM+(7s7G86Jo0|T&SeBd5ZDg;@eSNKN4EW*MX8xTgcD29`(nHD`_25;#-DeOu-|N~8LyXMQ+{DU7XuS}B3PgB z{t7edOw$Z_A-Jw!O5PeGtVgDq_HOfmx)3@`^nka ztO|X~nNG|pZp=0Kc&E-SqUAGOxXf^icO7+rwVhQ*P+k}RTwe=iXF}MGGjXJWr*4a7ieChf0DitdS`mKG-A{Gb6!bTDr zd}X$Til6fm+dP|uK_uymYm2Nsyy=I3_P zPEHRDVa8c zh6d9wZ|~JuOLh^67BTwhL^TtR(AJ*T-&OB`d$@^ca=C7wr38@pb(aa?o8q3JFI`fy zum4PmvU)H7M>Z_i_#UGSS&8M#uzn?apB7ymS<2#|L9{cPbkq_3jIG6>g#dKxTkaX@ zmS$fJ2O{$YmTD`XJA7U@E%2b8P{WU7V1s}6YO#GXCaIo|hS6-G3>%lzdxpw#ZYxWi zSSu<;WSs6}>S(M@IfgE#C22=BCnps=AU6FkehEJ>PGD!DW6e=ZC3>hx;`}?0)%};& zc07Y>h6lS1Yqon;l`r-!VuQh=iF#Aj zrsuRq8zZOq&k1c!b9HKm3TcR96%R(fZAY&sgxb0~O{H$;p1q&PO!7!z(#vs1=`QMj z5p}OIO~<|?M{SrznUN;sbjaU7HG8XSIC%S7!qFsqybbsGw3cT6L{nHH6C;iUV#I2^pm5Z8@I;?ZnV$=53s(nHLoulF-y3+RTN1L(U>&0KUsfjB$Y=hEum5;;D zoCkaHx*z_?ruRIGZo$?oeezkdeB};(rO4$#7oKfbJl@h0(+OFCGxI})>eNq|kcjOy z&3($id^e(=;`MbVCl;(Y#L>h`RodJ~4(hP14IbaTA$a8-qpLIbYB%?U`GkxH=kA`t zjasa>?z)aJFWD0u%4{FLQV5t7-DWQ{;BMx0R4`KD0iG9;7&I$tuCP|b#J@a!aWWOP z1|l}e?Dlv`O;_E@8Yg5Vy)-u9SaBx?SC*aIk}1zZU_IU>T?;~YPUiVMoXe7J;8+%y z!glYwrcLjV_@_}WUv4TiJ!9}s3kV^?JtHPEAdw|W=gXrZvh{Ch=*f-SBG^0#NaRa% z&>MsYYCAFzg}Bo3nD?AM8;rM>bbMSoN!ThSI~;Sy79?LTWKfoR6^UhU{ez>__3Ybd zfrA>YqsI&jK5tP;$HVyS(P(=>)Ggk^Q_#w>G}g*$Q)fy&R$g;k?3(c^30esjqupjd?+M# zkv5HCnM-)qVZ_zQSL=N;Wq#^p(j1oGJ2MO3x+5tnZcPkpu z#2O`iFG`v(0o(vh8hC4==*ZP=)-qh;;KxD=JhubT`^X_TXK!U<0S#u%$m-0VTp6S7 zvCyg{61kP)c z4dbH=Nl{hvO_z$YZAm(-%un<$obu=?84&FQJ7iZSq41zwp0TJ zO7T-@MpfT1vmWFfIBqy>mmO2j)0o46_3oY4pX`WDnI`c0%$(e3gi0vEvA6ujQ99s? zyb8-39d2i+4t?iQGC>(fgVF#M?n|P6%H(M_G|9*#J2AZVeG&6!xobFu|h4DMfuOH}6r)tj1H$FHRn7ksv z9P}*?^eGWe*%?h7;nXZ?Gup>_t=pMjwyT|H77j!Gr6*_el|1sCn+ zF!+W0W4q6z+79J)!|=MR;G@UQC9BGoduaQIDYPl3Dd|PRiEwDc?^@pwv1>Sc)bIiPLID>v;IQxsbqkl~?GZu(VM4{UHI)}p~H3yg`)v&r(Hnb8)fSvpo8Hyy=6C%`te zLB6KxEyszZz{ZDnJ~vz_wRlVLss^9AeB!BkqCUoog;q-Cl=tycy>DC3P9@f|SG1BeuQzb@e(rEN$p+Dgc;3g)n@~k+I688EuxlnwMJ-oWged}g zN>e~oVLXLPa8Wf}ZK_nw(!sGNJ$G?QLG!Vo zkUEcai>@b*l3}~Z{)G9K@wb&?v$vIdG9d5fs$=sdn(Wn?kj|KIp7^Vf$M7I(U-)}< z?AF|pJTalA_0!m9Pe=PT-H=D$HpK~Ss*LPkwDT~wc%jbl zAqb6Jug<85{1qC}Jztbhh)ED0d@@sZoFTId1*Xoum6OZE&6Y|jeXQ0IJ4_85oY|`S@`2MfHm+Q)Lku~R_0&6 z`y>Sg4Yq6O2LBEk{CXdF4Lr46Oa9}(d?%d%f;xQ&7scFv3F?qS;&PT>GN1m-cZqO_ z71(){tM+fKKt}M?**Ee*P~i8!{qeg%!JveF_45DX75(y!)S!e1kcYVaSHpk%O@|Ph z5WZdsY3%(Da{PHy4oGBy*AyPp{>yi}5T(*(+-^?guZY#}=#?du4#3U!k^7hLmY_u4 zVbyv|@&D9}*I-J5r!p$t&iI$_{_h*evl|yKoSlVqaVlQ8KsEzKDOblZ`b>4E_%;xy zfn@K*$)Z>&dwl5epN6yGhGD7)wFLIbkjXw14Dk^W%|?^KA0Vfc^+-)_yNQQg&VuW6 ze6)6Ybi4<+ZtE@+`=tRL<@?*&HtU=qjvl|g(mB3XSc||*<2_lIxrKs?jz!A<`ohIu zZ}_jF^rFHT?q|OF^U=>YpHNX@jHv(Z=ZTLl3J4j8(vtu61nB3eu-xAd{Qa2!{PaIP z@Sh&|PY?Y6*aJ9NWq|xioj=%Trwkw{E#$uREW1~ogMZt`*B9I|Fn9nPd~V!nJ+7-p zcFZVDUV)}JO(JHg>B*_p8;>Bq_t&+$P=MHi^$@^9TseHyn=Zu!GUQ19TA{hWY1*Cz|4vFg3$`iFCJN-*w?_Vcs-AG7`1<^E2aI zfblQtEVsV*Po+>{Z)?%O3;Q07g0YLquzBx$$GwsDhg5F~H+_Y-F@9G?L?lrYoJG$1 z(=D~A-hN0Ee9f=veoWnC-0#hHMuyG0Gq;cSXIpVsxQq(RvE0iR5AcYmXTv_wyeLqA zreZJhr(#-t;1_TC-oEkpLU8196Bmp1!vRL0^FJ+vfz9iZ4!_i8*rhk}t1A>6;;1k& zhGIEolY$0)64obz|2*pHWnv%`X$cT1ZG9DNo+2?RWYJeIqPSd+_fAY|ck4SO_G15Q zBfWoeI#D^Xml248pPYe3+NDT(=8fsd(#&@DeA$U~{I3m{ziR_i?UIKi&!y&?7oS0e zQ(R6z$N!la{IZD?0nR5S?0nqa2uD_7Su*~_yw{0IOn!eA_aT(Dfi} zxzS*r)ZKvG@B!W|p|t>~-0vp4pJl%OPI{%&eBs-qpG&>hH69#+`klqzR)C9yPs{u@ zs9Fb?>CHJ0tnL}!ttv3TYXSP!D}leA`p-V!g@wh+`T%W5uImA*Qhx^=Op76e8W;BO zqAr3DzGz#YaQU8XkJW-}T@q76i8cj5PS*z!<#zZ_V{;RETZkPTU0` z1R8fV?5EsaJzM@vXLPLla{&HB^H9I-?)?TIy4)XD#pUj6+q-R-Eu>^v~a527&eGCFzm&dzt)ft^0FmLLcJwx&G7T&;uU`gC>%3;YI(`rt`*$ zsDoB58Au=fRX;Ef>##7;@jVuF4Z&>wyFPbqKu>;35Lo@E|IY_Z=pN|P|3j63i(?OT zm0JN=MNz*k@;T68Un`sq77GV59ogbBY^JX?GxdK}x{mNZsbS#XgMrao*Py^cz4kBuvU>Np|jVId9$J z7DNp}CY&J?f=?X7cFgRTRVQ9Peau8S@_WMKfK?=Z;XC8`z;U5?5?Vej2e_an)`JEn zSCC?AWoQNSM(K`hq$RMdX$Po=!U zkW$U-GC?2Qz32|1IAQ^0BrZG+@s)c5*|C+WSe;pvGZ=zd00Ekss%*a9d=e@TL`vG& zn0{9cRi6R$XmJoP^nn&CDnwo=1S}ALqeW`?rg!1-R%^h}zE${;?8~FxvIp?DaDZA#Em{Uz z21)|}7t*mzAApRRI8Jk{u0y#Ns#voX7PnuK0Fl+_fben-8&hF3`HUn>R6lcW`F!*i zkRL9<-`fPl*=qdx0zi(AC>~JBqA%`~8SA&xNUwG~s3*xB#3K);3TVbJiM116QH_uI; z&SkcM4N^$WVwz*by&ZpcVCI#ulgaJ->Q4wb+JHUu`EN{lM^2Ka#3+Bx-RZXOG(li@Ih1hl^k`jECjF=R z)V~7E;L58zM3;bhoPXtBdpyT9=d$jk`Gkx;x%^9Jlh6JX%5 z?2uJo2EgZp;0IIy@G=LKi1Iv``|@G}NRB73aI-g!RdYoj?K+MVvGyI^`=kU!@3XX$ zhUzJ{4b@zB`*+Rs0Gy5v#PW>#OWeTZ>)(XbwwZvzQ#VL}K$V0}D+O_b4xl%d*jd`h zs|B!8W)GMEA0^m0mC;DHV_(gxUEVy?Bhq<6h-9g%BZR$b3MQW)lthYm8d%O#@is_9 zKxn^sXA}W1tpWnWc_`tStxwef6J)SyN1qc&$B21k-ap?MwL&59%z5u~9JHut-PgPc zt$LA#5!EK5TqHbXCFlKbxKBBjXosM7V8*B*Yq>y0;ZYloX?p!|r zsy3SK%F?C>`>ju@Lr6YmT#+p98N^rr2;TV21)2fOiR^maK?u=&JuJF1df-s zrf#^EMF%9h@@6%G8h?EYAzblmf(Uj#Q<}EKUd7vziE!n`+&WF?J6ZG>(m&ML(SsV{NyKo;bJ`}3Q?^=OIek%V@b1-d}9wZ%1! zh~wcn&FXZSXl+4wLQK75Ft}r_JSky`5Z0cAf%Mm4FB*#9f;LD@#3#h7=FzKBI!TP6 z3&OWY4<>N5_h=UzApxfv$E=~O#By`T=f|_ud>L$%7m!;KwLBRy<76N9^Nod5jh&HMyk- zW*l*7jjWn1UB|bCL8d2Xypvgv*xwZgJ|82Dsw;tP!{T^NFF~Z2yn?(K_2jdS%7tMc;CD|P; zR?N-s-K@}9Mn>{)M{GJzpKbZRq7o2yvH8x_u8hr`nv+OYf3xtLwy5i>E3gczyzBdl z9+G`yI}g|S6%0R#Jv5AzXY$2tXl6RtmDn@Z*pA-vx2sCY&5VAiRTYEm@SovXypFG5 zTp?s9Z=~wsq&xQklkZc+r)=tl?EUq%g07YBK<$SPV|lF9$0j9tLl5;8!ZWx-bg3dm zjnH5iQjp5lOQ%!OP9No6>E?}u!zvR1{?-HgVRjb98Gum1AB?9LNUABY6t9qy%yQt;b*TZP%p{7%9^GB z;mFm>UihKm0@n{PY-6uzWtj=?;`E^e^^9J~pWgKv{#SA9^7<~53*;@^+GHq+m| zT+Bc8J1nL3) zT?5jnd%3^%AVs-JI9`g<(9jmy>E~kh>EX^jI_{6zwWo)Afc=xuJlOfUQ}>Mrro_{_ z=Mk|1m`;nuf(?Ke!>fP)$^N^9_9PtH)$T4#)t)rKpK%z?Pc1Km1~ooK`}bb#68Xyz znJ@u2dR406(mm*WL)K-i=Dgp68q_Q?3H;81D0?O{{>EMd`M2U|N)bpzLai?REpUyq z#^o5nB_%H}dpQc3b#1rW8R0Z?dL;=BjCw$^hO=*SF7xI8= zqdDJuXDd=PO1!HpHtR)|-h+oG(}6OWT1f?E&5U~E4~r3`4t2I~RELUZd0+4=c6g%+HT`yZx{o!g`k zen&c8ndS;=fodUFq7G@F?3O@+g)7ANiN_nBeDnx#t5|hwWRoA;@Fzeafa{dZJ+_m2 zy(7W_?$r^ax}Gg&hkItSejUGXHSz6rTA-li6tAEM&KYzNt;t~Ry-6F>X!&LiLbkbN zEtNYRpVd0AFQ`Hml4R_`yl`4%GwlY}lHuu6g7p1x<(eYpHQ+E?PP$+5$H3b<1w*33 zvhtD9^fC+&c*_hn&o+W90PeqiY519xN4BH|ICS(?$_0Q;IyVfkMY4Lu?hT55^}rM} z3U}5Fm(FntnXR7JO=x%F{TU@lAqR~Qdzcz`-gq*1!0stKEUt9G(R6#j?gTEC61V9c zR{N6ZTI<>db2B?GaK#rCkhYXjROg*nd3wW>I^O<8+uq@0yRN|1c6pkm6P6Dz zcKa2XfCeOgIUo^nSBoy~HxcAMU!VCIC%~)(_Txuq;2vbQ>potN2WRS+M}`8|twhsdF03Eh=fq5-NJ>2% zsP4UN7T%A=l7->Rth3i_`Y~5?VMl_SpO|+QB^NDN1cdy#w{<>7)Mbl$5J9&)=>ey| zKNmW~JaRBNDQA1@(HGTE&B88HTe#CB7J=YFa8imXr)7glpyMd0tLpeecPSbD{xuN%Il(j{fvmf@GHLal!molTp8B1y_1{BBNQbdy4aLH? z8Kd+#wqB;Ts7R&|N;WaGw1>85cC|<9;MB##z8rP$!LGg`zo?+^hsAYy8AXLp-~dhM zSi-zJtGMQO#v34H^`fOBS3mEy^}1Amr+4_8#%;qw-Zccv6BP2yf)dAGHjCLljST4*S{f(9V+koIlqpm* zbWGi$i`q_(471w0*>a&Vx|ye-H=NPW5(6v)X}0`5Aq1jk*4ZHw8WU6!Qk zrSAs4V#$h3F)ImJM!=Pd+Cq)3uIXhvH%uE7x3E}`*>_#op^o99Kv|p$bhwVqGmE>q zb5+UmJ7pHxRX{^BMCd>AD_=!R53K6l)QsT(yGX{wJKz+4$ z@7euZc@^p6d+o>AzhZ&XK8auCe}-x2`Q3psX%2M%P#lF+Vj*~ztwv{tRAJi!xYA99 z=v2ix=TYM)qu?;|B_v)UB%aFv#oMfKJAXRTgmhs4(vhmZ1X&7q&S#`TIDl>#8yv<^ znA`w%?J?vdID2#Xa|Kb(rB5lD{X#E&A^!wK;_v8zn*cM}#(*%u9ALxJ1Jag7wb-L$ zvMigQ2ea38ZcxJcXKwnz74&;*96%}8k$4db1Sn`Y%*C&_8xhQee&<^zySck*kn6Q( z>cBLD)1&+8{jVVDF$e@tib77!MHH*o{8S_>#n#h6j#m~CD^oc6wWdAofoX^U^nTY- zN^x0LswJL<#JBG-ang&ap&8Of1_pt1R+wA-eEI}Z-Z2OM=@z&KFWhE8IsNM0Lb(){ zB7Q2=_6;xSo<5bFSzw5>!b{s|DWga&XWW?Bqbb?16q7f%F%9KX=p5 zy)USK?nxl*i+MuR!+&1V&(VS0UoKFCZcarN(er5U`SOOm9uEx=_T5Q;dg;G~eHYNd z!3LpbMDyp#=Wp&h0tc@vfgA?hjB}^yR~Rpa0$BDqtJGA;vj6jy)&e*a3%U8Q2LAnH kRP+!3sr>(<=a2c0rUnnzn_qiESJys!EGe8TsP*Q503~dQs{jB1 literal 0 HcmV?d00001 diff --git a/proposals/2025-03-25_otel_delta_temporality_support.md b/proposals/2025-03-25_otel_delta_temporality_support.md index d931dcb..0b6380c 100644 --- a/proposals/2025-03-25_otel_delta_temporality_support.md +++ b/proposals/2025-03-25_otel_delta_temporality_support.md @@ -123,9 +123,12 @@ For the initial implementation, there should be a documented warning that deltas No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. Delta metrics will be filtered out from metrics being federated. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. + ### Querying deltas -`rate()` and `increase()` will be extended to support delta metrics too. If the `__type__` starts with `delta_`, execute delta-specific logic instead of the current cumulative logic. The delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. +*Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues guessing the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).* + +`rate()` and `increase()` will be extended to support delta metrics too. If the `__type__` starts with `delta_`, execute delta-specific logic instead of the current cumulative logic. For consistenct, the delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. `irate()` will also be extended to support delta metrics. @@ -133,45 +136,31 @@ Having functions transparently handle the temporality simplifies the user experi `resets()` does not apply to delta metrics, however, so will return no results plus a warning in this case. -While the intention is to eventually use `rate()`/`increase()` etc. for both delta and cumulative metrics, initially experimental functions prefixed with `delta_` will be introduced behind a delta-support feature flag. This is to make it clear that these are experimental and the logic could change as we start seeing how they work in real-world scenarios. In the long run, we’d move the logic into `rate()`. - -#### Guessing start and end of series - -The current `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. - -With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. Therefore the delta implementation of `rate()`/`increase()` will not try and guess when a series starts and ends. Instead, it will always assume the series is ongoing for the whole range and always extrapolate to the whole range. - -This could inflate the value of the series, which can be especially problematic during rollouts when old series are replaced by new ones. - -As part of the implementation, experiment with heuristics to try and improve this (e.g. if intervals between samples are regular and there are than X samples, assume the samples are continuously ingested and therefore a gap would mean the series ended). This would make the calculation more complex, however. +While the intention is to eventually use `rate()`/`increase()` etc. for both delta and cumulative metrics, initially experimental functions prefixed with `delta_` will be introduced behind a delta-support feature flag. This is to make it clear that these are experimental and the logic could change as we start seeing how they work in real-world scenarios. In the long run, we’d move the logic into `rate()` etc.. -#### `rate()`/`increase()` calculation +#### rate() calculation -When CT-per-sample is introduced, there will be more information that could be used to more accurately calculate the rate (specifically, the first sample can be taken into account). Therefore the calculation differs depending on whether there is a CT within the sample. +TODO: write some code to make this clearer -TODO: write code for these implementations to make it clearer +In general: `sum of all sample values / (last sample ts - first sample start ts)) * range`. If the start time of the first sample is outside the range, truncate the first sample. -*Without CT-per-sample rate()/increase()* +The current `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. -`(sum of second to last samples / (last sample ts - first sample ts))` (multiply by range if `increase()`) +With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. -We ignore the value of the first sample as we do not know when it started. +We could just not try and predict the start/end of the series and assume the series continues to extend to beyond the samples in the range. However, not predicting the start and end of the series could inflate the rate/increase value, which can be especially problematic during rollouts when old series are replaced by new ones. -*With CT-per-sample rate()/increase()* +Assuming `rate()` only has information about the sample within the range, guessing the start and end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has started ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series). -In this case, we don’t need to guess where the series started. As we have CT-per-sample, if a sample is before the range start, it can't be within in the range at all. We still cannot tell if there is a sample that overlaps with the end range, however. +As part of the implementation process, experiment with heuristics to try and improve this (e.g. if intervals between samples are regular and there are than X samples, assume the samples are continuously ingested and therefore a gap would mean the series ended). This would make the calculation more complex, however. -1. If the start time of the first sample is outside the range, truncate the sample so we only take into account of the value within the range: - * `first sample value = first sample value * (first sample interval - (range start ts - first sample start ts))` - * `first sample value ts = range start ts` -2. Calculate rate: `(sum of all samples / (last sample ts - range start ts))` - * Multiply by `range end ts - max(range start ts, first sample start ts)` for `increase()`. +With CT-per-sample, we do not need to predict the start of the series, as if a sample is before the range start, it can't be within in the range at all. -#### Non-extrapolation +#### Non-approximation -There may be cases where extrapolating to get the rate/increase over the selected range is unwanted for delta metrics. Extrapolation may work worse for deltas since we do not try and guess when series start and end. +There may be cases where approximating the rate/increase over the selected range is unwanted for delta metrics. Approximation may work worse for deltas since we do not try and guess when series start and end. -Users may prefer "non-extrapolation" behaviour that just gives them the sum of the sample values within the range. This can be accomplished with `sum_over_time()`. Note that this does not accurately give them the increase within the range. +Users may prefer "non-approximating" behaviour that just gives them the sum of the sample values within the range. This can be accomplished with `sum_over_time()`. Note that this does not accurately give them the increase within the range. As an example: @@ -189,10 +178,9 @@ One possible solution would to have a function that does `sum_over_time()` for d ### Handling missing StartTimeUnixNano -StartTimeUnixNano is optional in the OTEL spec ... -Keep it for OTEL compatibility -use spacing between intervals when possible -non-extrapolation +StartTimeUnixNano is optional in the OTEL spec. To ensure compatibility with the OTEL spec, this case should be supported. Also note that before implementing CT-per-sample, every sample will be missing StartTimeUnixNano. + +For functions that require an interval to operate (e.g. rate()/increase()), assume the spacing between samples is the ingestion interval when StartTimeUnixNano is missing. ## Alternatives @@ -243,10 +231,47 @@ Have a convention for naming metrics e.g. appending `_delta_counter` to a metric ### Querying deltas alternatives -TODO: these are the top ones, for more see ... +#### Lookahead and lookbehind of range + +The reason why `increase()`/`rate()` need extrapolation to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. + +To work out the increase more accurately, they would also have to look at the sample before and the sample after the range to see if there are samples that partially overlap with the range - in that case the partial overlaps should be added to the increase. + +This could be a new function, or changing the `rate()` function (it could be dangerous to adjust `rate()`/`increase()` though as they’re so widely used that users may be dependent on their current behaviour even if they are “less accurate”). + +With deltas, we don’t need to lookbehind if we had CT-per-sample, only lookahead. + +This would be a good long-term proposal for deltas (and cumulative metrics). + +#### Do sum_over_time() / range for delta `rate()` implementation + +Instead of trying to approximate the rate over the interval, just sum all the samples in the range and divide by the range for `rate()`. + +For cumulative metrics, just taking the samples in the range and not approximating to cover the whole range is a bad approach. In the cumulative case, this would mean just taking the difference between the first and last samples and dividing by the range. As the first and last samples are unlikely to perfectly align with the start and end of the range, taking the difference between the two is likely to be an underestimation of the increase for the range. + +For delta metrics, this is less likely to be an underestimation. Non-approximation would mean something different than in the cumulative case - summing all the samples together. It's less likely to be an underestimation because the start time of the first sample could be before the start of the query range. So the total range of the selected samples could be similar to the query range. + +Below is an image to demonstrate - the filled in blue squares are the samples within the range, with the lines between the interval for which the sample data was collected. The unfilled blue squad is the start time for the first sample in the range, which is before the start time of the query range, and the total range of the samples is similar to the query range, just offset. + +![Range covered by samples vs query range](../assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png) + +As noted in [Non-approximation](#non-approximation), the actual range covered by the sum could still be different from the query range in the delta case. For the ranges to match, the query range needs to be a multiple of the collection interval, which Prometheus does not enforce. Also, this finds the rate between the start time of the first sample and the end time of the last sample, which won't always match the start and end times of the query. + +For users wanting this behaviour instead of the suggested one (approximating the rate/increase over the selected range), it is still possible do with (`sum_over_time()` / ``). + +Having `rate()`/`increase()`) do different things for cumulative and delta metrics can be confusing (e.g. with deltas and integer samples, you'd always get an integer value if you use `sum_over_time()`, but the same wouldn't be true for cumulative metrics with the current `increase()` behaviour). + +If we were to add behaviour to do the cumulative version of "sum_over_time", that would likely be in a different function. And then you'd have different functions to do non-approximation for delta and cumulative metrics, which again causes confusion. + +If we went for this approach first, but then updated `rate() to lookahead and lookbehind in the long term, users depending on the "non-extrapolation" could be affected. + +#### Convert to cumulative on query + +Delta to cumulative conversion at query time doesn’t have the same out of order issues as conversion at ingest. When a query is executed, it uses a fixed snapshot of data. The order the data was ingested does not matter, the cumulative values are correctly calculated by processing the samples in timestamp-order. + +No function modification needed - all cumulative functions will work for samples ingested as deltas. -rate() to do sum_over_time() -convert to cumulative on read +However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. ## Action Plan From a96b4359f74f5ac5eb2391dfd1f3ee7abace7c60 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Thu, 27 Mar 2025 16:02:45 +0000 Subject: [PATCH 07/34] Rename and cleanup Signed-off-by: Fiona Liao --- ...5-03-25_otel-delta-temporality-support.md} | 49 +++++++++++++------ 1 file changed, 33 insertions(+), 16 deletions(-) rename proposals/{2025-03-25_otel_delta_temporality_support.md => 2025-03-25_otel-delta-temporality-support.md} (89%) diff --git a/proposals/2025-03-25_otel_delta_temporality_support.md b/proposals/2025-03-25_otel-delta-temporality-support.md similarity index 89% rename from proposals/2025-03-25_otel_delta_temporality_support.md rename to proposals/2025-03-25_otel-delta-temporality-support.md index 0b6380c..13a7e42 100644 --- a/proposals/2025-03-25_otel_delta_temporality_support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -1,17 +1,22 @@ -## Your Proposal Title +# OTEL delta temporality support * **Owners:** * @fionaliao -TODO: add others from delta wg + * Initial design started by @ArthurSens and @sh0rez + * TODO: add others from delta wg * **Implementation Status:** `Not implemented` * **Related Issues and PRs:** - * `` + * https://github.com/prometheus/prometheus/issues/12763 * **Other docs or links:** - * `` + * [Original design doc](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.0) + * Additional context + * [OpenTelemetry metrics: A guide to Delta vs. Cumulative temporality trade-offs](https://docs.google.com/document/d/1wpsix2VqEIZlgYDM3FJbfhkyFpMq1jNFrJgXgnoBWsU/edit?tab=t.0#heading=h.wwiu0da6ws68) + * [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q) + * [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) This design doc proposes adding native delta support to Prometheus. This means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. @@ -66,8 +71,6 @@ Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. #### Does not handle sparse metrics well As mentioned in Background, sparse metrics are more common with delta. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. -TODO: would intermittent be a better word to describe this behaviour? - ## Goals Goals and use cases for the solution as proposed in [How](#how): @@ -83,21 +86,22 @@ This document is for Prometheus server maintainers, PromQL maintainers, and Prom ## Non-Goals * Support for ingesting delta metrics via other means (e.g. remote-write) -* Support for non-monotonic sums +* Support for converting OTEL non-monotonic sums to Prometheus counters (currently these are converted to Prometheus gauges) These may come in later iterations of delta support, however. ## How ### Ingesting deltas -When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. -TODO: start time nano injection -TODO: move the CT-per-sample here as the eventual goal +When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. + +There is an effort towards adding CreatedTimestamp as a field for each sample ([PR](https://github.com/prometheus/prometheus/pull/16046/files)). This is for cumulative counters, but can be reused for deltas too. When this is completed, if `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. -If `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. The CreatedTimestamp field does not exist yet, but there is currently an effort towards adding it for cumulative counters ([PR](https://github.com/prometheus/prometheus/pull/16046/files)), and can be reused for deltas. Having the timestamp and the start timestamp stored in a sample means that there is the potential to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. +CT-per-sample is not a blocker for deltas - before this is ready, `StartTimeUnixNano` will just be ignored. ### Chunks + For the initial implementation, reuse existing chunk encodings. Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. If a value in a bucket drops, that counts as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Therefore a new counter reset hint/header is required, to indicate the cumulative counter reset behaviour for chunk cutting should not apply. @@ -168,7 +172,7 @@ As an example: * S2: StartTimeUnixNano: T2, TimeUnixNano: T4, Value: 1 * S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 -And `sum_over_time() was executed between T1 and T5. +And `sum_over_time()` was executed between T1 and T5. As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. @@ -186,9 +190,13 @@ For functions that require an interval to operate (e.g. rate()/increase()), assu ### Ingesting deltas alternatives -#### CreatedTimestamp per sample +#### Inject zeroes for StartTimeUnixNano + +[CreatedAt timestamps can be injected as 0-valued samples](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection). Similar could be done for StartTimeUnixNano. + +CT-per-sample is a better solution overall as it links the start timestamp with the sample. It makes it easier to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. -If `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. The CreatedTimestamp field does not exist yet, but there is currently an effort towards adding it for cumulative counters ([PR](https://github.com/prometheus/prometheus/pull/16046/files)), and can be reused for deltas. Having the timestamp and the start timestamp stored in a sample means that there is the potential to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. +If CT-per-sample takes too long, this could be a temporary solution. #### Treat as gauge To avoid introducing a new type, deltas could be represented as gauges instead and the start time ignored. @@ -273,9 +281,18 @@ No function modification needed - all cumulative functions will work for samples However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. +## Known unknowns + +### Native histograms performance + +To work out the delta for all the cumulative native histograms in an interval, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the interval, all of the buckets in all of the samples need to be summed, which is less efficient at query time. + ## Action Plan The tasks to do in order to migrate to the new idea. -* [ ] Task one -* [ ] Task two ... +TODO: break down further + +- [ ] Implement type and metadata proposal +- [ ] Add delta ingestion + `delta_` functions behind new feature flag +- [ ] Merge `delta_` functions with cumulative equivalent From 467e8d849f16e7ac06d54494c086ef234cb38fd3 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Thu, 27 Mar 2025 16:06:18 +0000 Subject: [PATCH 08/34] Add slack channel Signed-off-by: Fiona Liao --- proposals/2025-03-25_otel-delta-temporality-support.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 13a7e42..0bf54ad 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -13,6 +13,7 @@ * **Other docs or links:** * [Original design doc](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.0) + * [#prometheus-delta-dev](https://cloud-native.slack.com/archives/C08C6CMEUF6) - Slack channel for project * Additional context * [OpenTelemetry metrics: A guide to Delta vs. Cumulative temporality trade-offs](https://docs.google.com/document/d/1wpsix2VqEIZlgYDM3FJbfhkyFpMq1jNFrJgXgnoBWsU/edit?tab=t.0#heading=h.wwiu0da6ws68) * [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q) From 18ee9694228f50796fb2fd3da723835bfa153dee Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Thu, 27 Mar 2025 19:51:42 +0000 Subject: [PATCH 09/34] Clarify interval calculation for CT vs no CT Signed-off-by: Fiona Liao --- ...25-03-25_otel-delta-temporality-support.md | 20 ++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 0bf54ad..414bab4 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -129,6 +129,12 @@ No scraped metrics should have delta temporality as there is no additional benef Delta metrics will be filtered out from metrics being federated. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. +### Handling missing StartTimeUnixNano + +StartTimeUnixNano is optional in the OTEL spec. To ensure compatibility with the OTEL spec, this case should be supported. Also note that before implementing CT-per-sample, every sample will be missing StartTimeUnixNano. + +For functions that require an interval to operate (e.g. `rate()`/`increase()`), assume the spacing between samples is the ingestion interval when StartTimeUnixNano is missing. + ### Querying deltas *Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues guessing the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).* @@ -147,7 +153,9 @@ While the intention is to eventually use `rate()`/`increase()` etc. for both del TODO: write some code to make this clearer -In general: `sum of all sample values / (last sample ts - first sample start ts)) * range`. If the start time of the first sample is outside the range, truncate the first sample. +In general: +* If CT-per-sample is available: `sum of all sample values / (last sample ts - first sample start ts)) * range`. If the start time of the first sample is outside the range, truncate the first sample. +* If CT-per-sample is not available: `sum of second to last sample values / (last sample ts - first sample ts)) * range`. We skip the value of the first sample value as we do not know its interval. The current `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. @@ -157,6 +165,10 @@ We could just not try and predict the start/end of the series and assume the ser Assuming `rate()` only has information about the sample within the range, guessing the start and end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has started ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series). +To calculate the interval: +* If CT-per-sample is available, use the average of the intervals of the samples (i.e. TimeUnixNano - StartTimeUnixNano). +* If CT-per-sample is not available, use the average spacing between samples. + As part of the implementation process, experiment with heuristics to try and improve this (e.g. if intervals between samples are regular and there are than X samples, assume the samples are continuously ingested and therefore a gap would mean the series ended). This would make the calculation more complex, however. With CT-per-sample, we do not need to predict the start of the series, as if a sample is before the range start, it can't be within in the range at all. @@ -181,12 +193,6 @@ As the samples are written at TimeUnixNano, only S1 and S2 are inside the query One possible solution would to have a function that does `sum_over_time()` for deltas and the cumulative equivalent too (this requires subtracting the latest sample before the start of the range with the last sample in the range). This is outside the scope of this design, however. -### Handling missing StartTimeUnixNano - -StartTimeUnixNano is optional in the OTEL spec. To ensure compatibility with the OTEL spec, this case should be supported. Also note that before implementing CT-per-sample, every sample will be missing StartTimeUnixNano. - -For functions that require an interval to operate (e.g. rate()/increase()), assume the spacing between samples is the ingestion interval when StartTimeUnixNano is missing. - ## Alternatives ### Ingesting deltas alternatives From 2a67dfc6661daa89a99e4a4b11ac7526f73128db Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 15 Apr 2025 22:17:35 +0100 Subject: [PATCH 10/34] Update proposal * Simplified proposal - moved CT-per-sample to possible future extension instead of embedding within proposal * Changed proposal to have a new `__temporality__` label instead of extending `__type__` - probably better to keep metric type concept distinct from metric temporality. This also aligns with how OTEL models it. * Updated remote-write section - delta ingestion will actually be fully supported via remote write (since CT-per-sample is moved out of main proposal for now) * Moved temporary `delta_rate()` and `delta_increase()` functions suggestion to discarded alternative - not sure this is actually necessary if we have feature flag for temporality-aware functions anyway * Fleshed out implementation plan Signed-off-by: Fiona Liao --- ...25-03-25_otel-delta-temporality-support.md | 190 ++++++++++++------ 1 file changed, 123 insertions(+), 67 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 414bab4..74cf13b 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -6,7 +6,7 @@ * Initial design started by @ArthurSens and @sh0rez * TODO: add others from delta wg -* **Implementation Status:** `Not implemented` +* **Implementation Status:** `Partially implemented` * **Related Issues and PRs:** * https://github.com/prometheus/prometheus/issues/12763 @@ -27,13 +27,13 @@ Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167[). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. -Tthe cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. +The cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. Prometheus' goal of becoming the best OTEL metrics backend means we should support delta metrics properly. We propose to add native support for OTEL delta metrics (i.e. metrics ingested via the OTLP endpoint). Native support means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. -### Delta datapoints +### OTEL delta datapoints In the [OTEL spec](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality), like cumulative metrics, a datapoint for a delta metric has a `(start,end]` time window. However, the time windows of delta datapoints do not overlap. @@ -79,6 +79,8 @@ Goals and use cases for the solution as proposed in [How](#how): * Allow OTEL delta metrics to be ingested via the OTLP endpoint and stored as-is * Support for all OTEL metric types that can have delta temporality (sums, histograms, exponential histograms) * Queries behave appropriately for delta metrics +* Support for ingestion delta metrics via remote write + * The main focus of the proposal is on OTLP ingestion, but remote write deltas end up being supported too. ### Audience @@ -86,7 +88,7 @@ This document is for Prometheus server maintainers, PromQL maintainers, and Prom ## Non-Goals -* Support for ingesting delta metrics via other means (e.g. remote-write) +* Support for ingesting delta metrics via other means (e.g. replacing push gateway) * Support for converting OTEL non-monotonic sums to Prometheus counters (currently these are converted to Prometheus gauges) These may come in later iterations of delta support, however. @@ -97,9 +99,9 @@ These may come in later iterations of delta support, however. When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. -There is an effort towards adding CreatedTimestamp as a field for each sample ([PR](https://github.com/prometheus/prometheus/pull/16046/files)). This is for cumulative counters, but can be reused for deltas too. When this is completed, if `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. +For the initial implementation, ignore `StartTimeUnixNano`. -CT-per-sample is not a blocker for deltas - before this is ready, `StartTimeUnixNano` will just be ignored. +To ensure compatibility with the OTEL spec, this case should be supported, however. A way to preserve `StartTimeUnixNano` is described in the potential future extension, [CT-per-sample](#ct-per-sample). ### Chunks @@ -111,17 +113,21 @@ Currently the counter reset behaviour for cumulative native histograms is to cut We need to be able to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. -Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). The `__type__` label will be extended with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) +Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). An additional `__temporality__` label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality is assumed to be cumulative. + +When ingesting a delta metric via the OTLP endpoint, the `__temporality__="delta"` label will be added. -When ingesting a delta metric via the OTLP endpoint, the type will be added as a label. +Not all metric types should have a temporality (e.g. gauge). For those types, a `__temporality__` label will not be added by the OTLP endpoint. -A downside is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)”` or `__type__=~”delta_.*”` would have to be used. However, this does not seem like a particularly necessary use case to optimise for. +It is possible to manually add the `__temporality__` label to a metric with a non-temporality type. The Prometheus query engine will disregard this label for such metric types. + +Initially, `__temporality__="cumulative"` will not be added to cumulative metrics ingested via the OTLP endpoint to avoid unnecessary churn for exisitng cumulative metrics and potential disruption for users who might not be expecting this new label. ### Remote write -Remote write support is a non-goal for the initial implementation to reduce its scope. However, the current design ends up partially supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. However, there is currently no equivalent to StartTimeUnixNano per sample in remote write. +Remote write support is a non-goal for this proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. -For the initial implementation, there should be a documented warning that deltas are not _properly_ supported with remote write yet. +There is currently no equivalent to StartTimeUnixNano per sample in remote write. However, the initial delta implementation drops that field anyway. ### Scraping @@ -129,17 +135,11 @@ No scraped metrics should have delta temporality as there is no additional benef Delta metrics will be filtered out from metrics being federated. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. -### Handling missing StartTimeUnixNano - -StartTimeUnixNano is optional in the OTEL spec. To ensure compatibility with the OTEL spec, this case should be supported. Also note that before implementing CT-per-sample, every sample will be missing StartTimeUnixNano. - -For functions that require an interval to operate (e.g. `rate()`/`increase()`), assume the spacing between samples is the ingestion interval when StartTimeUnixNano is missing. - ### Querying deltas *Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues guessing the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).* -`rate()` and `increase()` will be extended to support delta metrics too. If the `__type__` starts with `delta_`, execute delta-specific logic instead of the current cumulative logic. For consistenct, the delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. +`rate()` and `increase()` will be extended to support delta metrics too. If the `__temporality__` is `delta`, execute delta-specific logic instead of the current cumulative logic. For consistency, the delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. `irate()` will also be extended to support delta metrics. @@ -147,15 +147,9 @@ Having functions transparently handle the temporality simplifies the user experi `resets()` does not apply to delta metrics, however, so will return no results plus a warning in this case. -While the intention is to eventually use `rate()`/`increase()` etc. for both delta and cumulative metrics, initially experimental functions prefixed with `delta_` will be introduced behind a delta-support feature flag. This is to make it clear that these are experimental and the logic could change as we start seeing how they work in real-world scenarios. In the long run, we’d move the logic into `rate()` etc.. - #### rate() calculation -TODO: write some code to make this clearer - -In general: -* If CT-per-sample is available: `sum of all sample values / (last sample ts - first sample start ts)) * range`. If the start time of the first sample is outside the range, truncate the first sample. -* If CT-per-sample is not available: `sum of second to last sample values / (last sample ts - first sample ts)) * range`. We skip the value of the first sample value as we do not know its interval. +In general: `sum of second to last sample values / (last sample ts - first sample ts)) * range`. We skip the value of the first sample as we do not know its interval. The current `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. @@ -165,39 +159,35 @@ We could just not try and predict the start/end of the series and assume the ser Assuming `rate()` only has information about the sample within the range, guessing the start and end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has started ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series). -To calculate the interval: -* If CT-per-sample is available, use the average of the intervals of the samples (i.e. TimeUnixNano - StartTimeUnixNano). -* If CT-per-sample is not available, use the average spacing between samples. +To calculate the interval, use the average spacing between samples. -As part of the implementation process, experiment with heuristics to try and improve this (e.g. if intervals between samples are regular and there are than X samples, assume the samples are continuously ingested and therefore a gap would mean the series ended). This would make the calculation more complex, however. +Downsides: -With CT-per-sample, we do not need to predict the start of the series, as if a sample is before the range start, it can't be within in the range at all. +* This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). +* Harder to predict the start and end of the series vs cumulative. +* The average spacing may not be a good estimation for the ingestion interval, since delta metrics can be sparse. #### Non-approximation -There may be cases where approximating the rate/increase over the selected range is unwanted for delta metrics. Approximation may work worse for deltas since we do not try and guess when series start and end. - -Users may prefer "non-approximating" behaviour that just gives them the sum of the sample values within the range. This can be accomplished with `sum_over_time()`. Note that this does not accurately give them the increase within the range. +While `rate()` and `increase()` will be consistent with cumulative case, there are downsides as mentioned above. Therefore there may be cases where approximating the rate/increase over the selected range is unwanted for delta metrics. -As an example: +Users may prefer "non-approximating" behaviour that just gives them the sum of the sample values within the range. This can be accomplished with `sum_over_time()`, though this does not always accurately give the increase within the requested range. -* S1: StartTimeUnixNano: T0, TimeUnixNano: T2, Value: 5 -* S2: StartTimeUnixNano: T2, TimeUnixNano: T4, Value: 1 -* S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 +`sum_over_time()` does not work for cumulative metrics, so a warning should be returned in this case. One downside is that this could make migrating from delta to cumulative metrics harder, since `sum_over_time()` queries would need to be rewritten, and users wanting to use `sum_over_time()` will need to know the temporality of their metrics. -And `sum_over_time()` was executed between T1 and T5. +One possible solution would to have a function that does `sum_over_time()` for deltas and the cumulative equivalent too (this requires subtracting the latest sample before the start of the range with the last sample in the range). This is outside the scope of this design, however. -As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. +## Possible future extensions -`sum_over_time()` does not work for cumulative metrics, so a warning should be returned in this case. One downside is that this could make migrating from delta to cumulative metrics harder, since `sum_over_time()` queries would need to be rewritten, and users wanting to use `sum_over_time()` will need to know the temporality of their metrics. +### CT-per-sample -One possible solution would to have a function that does `sum_over_time()` for deltas and the cumulative equivalent too (this requires subtracting the latest sample before the start of the range with the last sample in the range). This is outside the scope of this design, however. +There is an effort towards adding CreatedTimestamp as a field for each sample ([PR](https://github.com/prometheus/prometheus/pull/16046/files)). This is for cumulative counters, but can be reused for deltas too. When this is completed, if `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. -## Alternatives +CT-per-sample is not a blocker for deltas - before this is ready, `StartTimeUnixNano` will just be ignored. -### Ingesting deltas alternatives +Having CT-per-sample can improve the `rate()` calculation - the ingestion interval for each sample will be directly available, rather than having to guess the interval based on gaps. It also means a single sample in the range can result in a result from `rate()` as the range will effectively have an additional point at `StartTimeUnixNano`. -#### Inject zeroes for StartTimeUnixNano +### Inject zeroes for StartTimeUnixNano [CreatedAt timestamps can be injected as 0-valued samples](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection). Similar could be done for StartTimeUnixNano. @@ -205,6 +195,19 @@ CT-per-sample is a better solution overall as it links the start timestamp with If CT-per-sample takes too long, this could be a temporary solution. +### Lookahead and lookbehind of range + +The reason why `increase()`/`rate()` need extrapolation to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. + +To work out the increase more accurately, they would also have to look at the sample before and the sample after the range to see if there are samples that partially overlap with the range - in that case the partial overlaps should be added to the increase. + +This could be a new function, or changing the `rate()` function (it could be dangerous to adjust `rate()`/`increase()` though as they’re so widely used that users may be dependent on their current behaviour even if they are “less accurate”). + + +## Alternatives + +### Ingesting deltas alternatives + #### Treat as gauge To avoid introducing a new type, deltas could be represented as gauges instead and the start time ignored. @@ -220,6 +223,7 @@ Functions will not take into account delta-specific characteristics. The OTEL SD This also does not work for samples missing StartTimeUnixNano. #### Convert to rate on ingest + Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixName` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. The difference is that there is no interval information in the metric name (like :rate1m) as there is no guarantee that the interval from sample to sample stays constant. @@ -240,27 +244,23 @@ A new `__temporality__` label could be added instead. However, not all metric types should have a temporality (e.g. gauge). Having `delta_` as part of the type label enforces that only specific metric types can have temporality. Otherwise, additional label error checking would need to be done to make sure `__temporality__` is only added to specific types. -#### Metric naming convention - -Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. - -### Querying deltas alternatives +#### Add delta `__type__` label values -#### Lookahead and lookbehind of range +Instead of a new `__temporality__` label, extend `__type__` from the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files) with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) -The reason why `increase()`/`rate()` need extrapolation to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. +A downside is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)”` or `__type__=~”delta_.*”` would have to be used. (However, this does not seem like a particularly necessary use case to optimise for.) -To work out the increase more accurately, they would also have to look at the sample before and the sample after the range to see if there are samples that partially overlap with the range - in that case the partial overlaps should be added to the increase. +Additionally, combining temporality and type means that every time a new type is added to Prometheus/OTEL, two `__type__` values would have to be added. This is unlikely to happen very often, so only a minor con. -This could be a new function, or changing the `rate()` function (it could be dangerous to adjust `rate()`/`increase()` though as they’re so widely used that users may be dependent on their current behaviour even if they are “less accurate”). +#### Metric naming convention -With deltas, we don’t need to lookbehind if we had CT-per-sample, only lookahead. +Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. -This would be a good long-term proposal for deltas (and cumulative metrics). +### Querying deltas alternatives #### Do sum_over_time() / range for delta `rate()` implementation -Instead of trying to approximate the rate over the interval, just sum all the samples in the range and divide by the range for `rate()`. +Instead of trying to approximate the rate over the interval, just sum all the samples in the range and divide by the range for `rate()`. This avoids any approximation which may be less effective for delta metrics where the start and end of series is harder to predict, so may be preferred by users. For cumulative metrics, just taking the samples in the range and not approximating to cover the whole range is a bad approach. In the cumulative case, this would mean just taking the difference between the first and last samples and dividing by the range. As the first and last samples are unlikely to perfectly align with the start and end of the range, taking the difference between the two is likely to be an underestimation of the increase for the range. @@ -270,15 +270,45 @@ Below is an image to demonstrate - the filled in blue squares are the samples wi ![Range covered by samples vs query range](../assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png) -As noted in [Non-approximation](#non-approximation), the actual range covered by the sum could still be different from the query range in the delta case. For the ranges to match, the query range needs to be a multiple of the collection interval, which Prometheus does not enforce. Also, this finds the rate between the start time of the first sample and the end time of the last sample, which won't always match the start and end times of the query. +The actual range covered by the sum could still be different from the query range in the delta case. For the ranges to match, the query range needs to be a multiple of the collection interval, which Prometheus does not enforce. Also, this finds the rate between the start time of the first sample and the end time of the last sample, which won't always match the start and end times of the query. + +Below are a couple of examples. + +**Example 1** + +* S1: StartTimeUnixNano: T0, TimeUnixNano: T2, Value: 5 +* S2: StartTimeUnixNano: T2, TimeUnixNano: T4, Value: 1 +* S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 + +And `sum_over_time()` was executed between T1 and T5. + +As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. + +**Example 2** + +* S1: StartTimeUnixNano: T0, TimeUnixNano: T5, Value: 10 + +`sum_over_time()` between T0 and T5 will get 10. Divided by 5 for the rate results in 2. + +However, if you only query between T4 and T5, the rate would be 10/1 = 1 , and queries between earlier times (T0-T1, T1-T2 etc.) will have a rate of zero. These results may be misleading. -For users wanting this behaviour instead of the suggested one (approximating the rate/increase over the selected range), it is still possible do with (`sum_over_time()` / ``). +For users wanting this behaviour instead of the suggested one (approximating the rate/increase over the selected range), it is still possible do with `sum_over_time()` / ``. -Having `rate()`/`increase()`) do different things for cumulative and delta metrics can be confusing (e.g. with deltas and integer samples, you'd always get an integer value if you use `sum_over_time()`, but the same wouldn't be true for cumulative metrics with the current `increase()` behaviour). +Having `rate()`/`increase()` do different things for cumulative and delta metrics can be confusing (e.g. with deltas and integer samples, you'd always get an integer value if you use `sum_over_time()`, but the same wouldn't be true for cumulative metrics with the current `increase()` behaviour). If we were to add behaviour to do the cumulative version of "sum_over_time", that would likely be in a different function. And then you'd have different functions to do non-approximation for delta and cumulative metrics, which again causes confusion. -If we went for this approach first, but then updated `rate() to lookahead and lookbehind in the long term, users depending on the "non-extrapolation" could be affected. +If we went for this approach first, but then updated `rate()` to lookahead and lookbehind in the long term, users depending on the "non-extrapolation" could be affected. + +One open question is if or how much the "accuracy" problem matters to users. Furthermore, in the case of the second example, where the query range is smaller than the ingestion interval, that may not be a very frequent problem. + +#### Do sum_over_time() / range for delta `rate()` implementation only when StartTimeUnixNano is missing + +Use the proposed logic (with interval-based approximation) when StartTimeUnixNano is set, but if it's missing, use sum_over_time() / range. + +With the initial implementation, all delta metrics will essentially have missing "StartTimeUnixNano" since that is discarded on ingestion and not available at query time. + +This also provides a way for users to choose between the approximating and non-approximating query behaviour by setting StartTimeUnixNano or not. However, the users querying the metrics may be different from the operators of the metrics pipeline, and therefore still not have control over the query behaviour. #### Convert to cumulative on query @@ -288,18 +318,44 @@ No function modification needed - all cumulative functions will work for samples However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. +#### Have temporary `delta_rate()` and `delta_increase()` functions + +While the intention is to eventually use `rate()`/`increase()` etc. for both delta and cumulative metrics, initially experimental functions prefixed with `delta_` will be introduced behind a delta-support feature flag. This is to make it clear that these are experimental and the logic could change as we start seeing how they work in real-world scenarios. In the long run, we’d move the logic into `rate()` etc.. + +This may be an unnecessary step, especially if the delta functionality is behind feature flags. + ## Known unknowns ### Native histograms performance -To work out the delta for all the cumulative native histograms in an interval, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the interval, all of the buckets in all of the samples need to be summed, which is less efficient at query time. +To work out the delta for all the cumulative native histograms in an range, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the range, all of the buckets in all of the samples need to be summed, which is less efficient at query time. + +## Implementation Plan + +### Milestone 1: Primitive support for delta ingestion + +Behind a feature flag (`otlp-native-delta-ingestion`), allow OTLP metrics with delta temporality to be ingested and stored as-is, with metric type unknown and no additional labels. To get "increase" or "rate", `sum_over_time(metric[] / )` can be used. + +Having this simple implementation without changing any PromQL functions allow us to get some form of delta ingestion out there gather some feedback to decide the best way to go further. + +PR: https://github.com/prometheus/prometheus/pull/16360 + +### Milestone 2: Introduce temporality label + +The second milestone depends on the type and unit metadata proposal being implemented. + +Introduce the `__temporality__` label, similar to how the `__type__` and `__unit__` labels have been added. + +Add `__temporality__="delta"` to delta metrics ingested via the OTLP endpoint (still under the `otlp-native-delta-ingestion` feature flag). + +### Milestone 3: Temporality-aware functions + +Update functions to use the `__temporality__` label. -## Action Plan +Update `rate()`, `increase()` and `idelta()` functions to support delta metrics. -The tasks to do in order to migrate to the new idea. +Add warnings to `resets()` if used with delta metrics, and `sum_over_time()` with cumualtive metrics. -TODO: break down further +Temporality-aware functions should be under a `temporality-aware-functions` feature flag. -- [ ] Implement type and metadata proposal -- [ ] Add delta ingestion + `delta_` functions behind new feature flag -- [ ] Merge `delta_` functions with cumulative equivalent +Also filter out metrics with `__temporality__="delta"` from the federation endpoint. From 74dc016a55a26f589c0b72a2579a744b8354a0c0 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 18 Apr 2025 17:07:19 +0100 Subject: [PATCH 11/34] Small updates, point out temporality label could be change depending on type and metadata Signed-off-by: Fiona Liao --- .../2025-03-25_otel-delta-temporality-support.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 74cf13b..292f788 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -41,7 +41,7 @@ The `end` timestamp is called `TimeUnixNano` and is mandatory. The `start` time ### Characteristics of delta metrics -Sparse metrics are more common for delta than cumulative metrics. While delta datapoints can be emitted at a regular interval, in some cases (like the OTEL SDKs), datapoints are only emitted when there is a change (e.g. if tracking request count, only send a datapoint if the number of requests in the ingestion interval > 0). This can be beneficial for the metrics producer, reducing memory usage and network bandwidth. +Sparse metrics are more common for delta than cumulative metrics. While delta datapoints can be emitted at a regular interval, in some cases (like the OTEL SDKs), datapoints are only emitted when there is a change (e.g. if tracking request count, only send a datapoint if the number of requests in the ingestion interval > 0). This can be beneficial for the metrics producer, reducing memory usage and network bandwidth. Further insights and discussions on delta metrics can be found in [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y), which describes Chronosphere's experience of adding functionality to ingest OTEL delta metrics and query them back with PromQL, and also [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q). @@ -80,7 +80,7 @@ Goals and use cases for the solution as proposed in [How](#how): * Support for all OTEL metric types that can have delta temporality (sums, histograms, exponential histograms) * Queries behave appropriately for delta metrics * Support for ingestion delta metrics via remote write - * The main focus of the proposal is on OTLP ingestion, but remote write deltas end up being supported too. + * The main focus of the proposal is on OTLP ingestion, but remote write deltas end up being supported too. (TODO: depends on type and unit metadata proposal) ### Audience @@ -123,9 +123,13 @@ It is possible to manually add the `__temporality__` label to a metric with a no Initially, `__temporality__="cumulative"` will not be added to cumulative metrics ingested via the OTLP endpoint to avoid unnecessary churn for exisitng cumulative metrics and potential disruption for users who might not be expecting this new label. +TODO: need type and unit metadata proposal to be done to confirm how a new temporality metadata label could be added + ### Remote write -Remote write support is a non-goal for this proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. +Remote write support is a non-goal for this proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. + +TODO: label setting depends on the type and unit metadata proposal, may need to be updated There is currently no equivalent to StartTimeUnixNano per sample in remote write. However, the initial delta implementation drops that field anyway. @@ -137,13 +141,13 @@ Delta metrics will be filtered out from metrics being federated. If the current ### Querying deltas -*Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues guessing the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).* +*Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues with sparse deltas, like making it harder to guess the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).* `rate()` and `increase()` will be extended to support delta metrics too. If the `__temporality__` is `delta`, execute delta-specific logic instead of the current cumulative logic. For consistency, the delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. `irate()` will also be extended to support delta metrics. -Having functions transparently handle the temporality simplifies the user experience - users do not need to know the temporality of a series for querying, and means queries don't need to be rewriten wehn migrating between cumulative and delta metrics. +Having functions transparently handle the temporality simplifies the user experience - users do not need to know the temporality of a series for querying, and means queries don't need to be rewriten when migrating between cumulative and delta metrics. `resets()` does not apply to delta metrics, however, so will return no results plus a warning in this case. @@ -203,7 +207,6 @@ To work out the increase more accurately, they would also have to look at the sa This could be a new function, or changing the `rate()` function (it could be dangerous to adjust `rate()`/`increase()` though as they’re so widely used that users may be dependent on their current behaviour even if they are “less accurate”). - ## Alternatives ### Ingesting deltas alternatives From 5ddeb56d324dcec4846d558b31553738314f6744 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Mon, 16 Jun 2025 22:49:17 +0100 Subject: [PATCH 12/34] Start updating post comments - Make clear experimental nature and opt-in - Detail two feature flags for ingestion Signed-off-by: Fiona Liao --- ...25-03-25_otel-delta-temporality-support.md | 104 +++++++++++------- 1 file changed, 64 insertions(+), 40 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 292f788..0cc6c94 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -19,7 +19,7 @@ * [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q) * [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) - This design doc proposes adding native delta support to Prometheus. This means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. +This design document proposes adding experimental support for OTEL delta temporality metrics in Prometheus, allowing them be ingested and stored directly. ## Why @@ -29,9 +29,9 @@ Therefore, delta metrics need to be converted to cumulative ones during ingestio The cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. -Prometheus' goal of becoming the best OTEL metrics backend means we should support delta metrics properly. +Prometheus' goal of becoming the best OTEL metrics backend means it should improve its support for delta metrics, allowing them to be ingested and stored without being transformed into cumulative. -We propose to add native support for OTEL delta metrics (i.e. metrics ingested via the OTLP endpoint). Native support means storing delta metrics without transforming to cumulative, and having functions that behave appropriately for delta metrics. +We propose some initial steps for delta support in this document. These delta features will be experimental and opt-in, allowing us to gather feedback and gain practical experience with deltas before deciding on future steps. ### OTEL delta datapoints @@ -67,7 +67,7 @@ State becomes more complex in distributed cases - if there are multiple OTEL col #### Values written aren’t the same as the values read -Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. call to get a useful result. However, it could be confusing when querying just the metric without any functions, the returned value is not the same as the ingested value. +Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. call to get a useful result. However, it could be confusing that when querying just the metric without any functions, the returned value is not the same as the ingested value. #### Does not handle sparse metrics well As mentioned in Background, sparse metrics are more common with delta. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. @@ -76,11 +76,10 @@ As mentioned in Background, sparse metrics are more common with delta. This can Goals and use cases for the solution as proposed in [How](#how): -* Allow OTEL delta metrics to be ingested via the OTLP endpoint and stored as-is -* Support for all OTEL metric types that can have delta temporality (sums, histograms, exponential histograms) -* Queries behave appropriately for delta metrics -* Support for ingestion delta metrics via remote write - * The main focus of the proposal is on OTLP ingestion, but remote write deltas end up being supported too. (TODO: depends on type and unit metadata proposal) +* Allow OTEL delta metrics to be ingested via the OTLP endpoint and stored directly. +* Support for all OTEL metric types that can have delta temporality (sums, histograms, exponential histograms). +* Allow delta metrics to be distinguished from cumulative metrics. +* Allow the query engine to flag warnings when a cumulative function is used on deltas. ### Audience @@ -88,10 +87,10 @@ This document is for Prometheus server maintainers, PromQL maintainers, and Prom ## Non-Goals -* Support for ingesting delta metrics via other means (e.g. replacing push gateway) -* Support for converting OTEL non-monotonic sums to Prometheus counters (currently these are converted to Prometheus gauges) +* Support for ingesting delta metrics via other, non-OTLP means (e.g. replacing push gateway). +* Advanced querying support for deltas (e.g. function overloading for rate()). Given that delta support is new and advanced querying also depends on other experimental features, the best approach - or whether we should even extend querying in any way - is currently unclear. However, this document does explore some possible options. -These may come in later iterations of delta support, however. +These may come in later iterations of delta support. ## How @@ -99,48 +98,75 @@ These may come in later iterations of delta support, however. When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. -For the initial implementation, ignore `StartTimeUnixNano`. - -To ensure compatibility with the OTEL spec, this case should be supported, however. A way to preserve `StartTimeUnixNano` is described in the potential future extension, [CT-per-sample](#ct-per-sample). +For the initial implementation, ignore `StartTimeUnixNano`. To ensure compatibility with the OTEL spec, this case should ideally be supported. A way to preserve `StartTimeUnixNano` is described in the potential future extension, [CT-per-sample](#ct-per-sample). ### Chunks -For the initial implementation, reuse existing chunk encodings. +For the initial implementation, reuse existing chunk encodings. + +Delta counters will use the standard XOR chunks for float samples. + +Delta histograms will use native histogram chunks with the `GaugeType` counter reset hint/header. Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. A (bucket or total) count drop is detected as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Additionally counter histogram chunks have the invariant that no count ever goes down baked into their implementation. `GaugeType` allows counts to go up and down, and does not cut new chunks on counter resets. + +### Delta metric type + +It is useful to be able to distinguish between delta and cumulative metrics. This would allow users to understand out what the raw data represents and what functions are appropiate to use. Additionally, this could allow the query engine or UIs displaying Prometheus data apply different behaviour depending on the metric type to provide meaningful output. -Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. If a value in a bucket drops, that counts as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Therefore a new counter reset hint/header is required, to indicate the cumulative counter reset behaviour for chunk cutting should not apply. +There are two options we are considering for how to type deltas. We propose to add both of these options as feature flags for ingesting deltas. -### Distinguishing between delta and cumulative metrics +1. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges. -We need to be able to distinguish between delta and cumulative metrics. This would allow the query engine to apply different behaviour depending on the metric type. Users should also be able to see the temporality of a metric, which is useful for understanding the metric and for debugging. +2. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas as a new delta "type", using a new `__temporality__` label to explicitly mark metrics as delta. -Our suggestion is to build on top of the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files). An additional `__temporality__` label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality is assumed to be cumulative. +In the documentation, we would highlight that the gauge option is more stable, since it's a pre-exisiting type and has been used for delta-like use cases in Prometheus already, while the temporality label option is very experimental and dependent on other experimental features. -When ingesting a delta metric via the OTLP endpoint, the `__temporality__="delta"` label will be added. +Below we explore the pros and cons of each option in more detail. -Not all metric types should have a temporality (e.g. gauge). For those types, a `__temporality__` label will not be added by the OTLP endpoint. +#### Treat as gauge -It is possible to manually add the `__temporality__` label to a metric with a non-temporality type. The Prometheus query engine will disregard this label for such metric types. +Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. -Initially, `__temporality__="cumulative"` will not be added to cumulative metrics ingested via the OTLP endpoint to avoid unnecessary churn for exisitng cumulative metrics and potential disruption for users who might not be expecting this new label. +Pros +* Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. +* Prometheus already implicitly uses gauges to represent deltas. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. -TODO: need type and unit metadata proposal to be done to confirm how a new temporality metadata label could be added +Cons +This could be confusing as gauges are usually used for sampled data (for example, in OTEL: "Gauges do not provide an aggregation semantic, instead “last sample value” is used when performing operations like temporal alignment or adjusting resolution.”) rather than data that should be summed/rated over time. -### Remote write +TODO: cover temporality label +### Remote write +Not a goal but sort of supported with temporality label option +ingestion and forwarding - forwarding will contain labels +otel is translated into remote write request, that'll be propagated Remote write support is a non-goal for this proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. TODO: label setting depends on the type and unit metadata proposal, may need to be updated There is currently no equivalent to StartTimeUnixNano per sample in remote write. However, the initial delta implementation drops that field anyway. +TODO: say remote write will be supported via adding a temporality label ### Scraping No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. Delta metrics will be filtered out from metrics being federated. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. +TODO: not always the case - instead of filtering update +metadata counter + __temporality__ -> can miss the temporality + +### TODO: non-monotonic sums + +### TODO: + +### Prometheus OTEL receiver +to be able to convert back to delta ### Querying deltas +TODO: add warnings and that's it. move the rest to potential future extensions +The existing `sum_over_time()` function can be used to aggregate a delta metric over time, and `sum_over_time(metric[]) / ` can be used for the rate. TODO: link + + *Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues with sparse deltas, like making it harder to guess the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).* `rate()` and `increase()` will be extended to support delta metrics too. If the `__temporality__` is `delta`, execute delta-specific logic instead of the current cumulative logic. For consistency, the delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. @@ -181,6 +207,8 @@ Users may prefer "non-approximating" behaviour that just gives them the sum of t One possible solution would to have a function that does `sum_over_time()` for deltas and the cumulative equivalent too (this requires subtracting the latest sample before the start of the range with the last sample in the range). This is outside the scope of this design, however. +TODO: mixed samples + ## Possible future extensions ### CT-per-sample @@ -198,8 +226,10 @@ Having CT-per-sample can improve the `rate()` calculation - the ingestion interv CT-per-sample is a better solution overall as it links the start timestamp with the sample. It makes it easier to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. If CT-per-sample takes too long, this could be a temporary solution. +TODO: also mention space and performance of additional sample ### Lookahead and lookbehind of range +TODO: move to querying section The reason why `increase()`/`rate()` need extrapolation to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. @@ -207,21 +237,16 @@ To work out the increase more accurately, they would also have to look at the sa This could be a new function, or changing the `rate()` function (it could be dangerous to adjust `rate()`/`increase()` though as they’re so widely used that users may be dependent on their current behaviour even if they are “less accurate”). -## Alternatives - -### Ingesting deltas alternatives - -#### Treat as gauge -To avoid introducing a new type, deltas could be represented as gauges instead and the start time ignored. +## Discarded alternatives -This could be confusing as gauges are usually used for sampled data (for example, in OTEL: "Gauges do not provide an aggregation semantic, instead “last sample value” is used when performing operations like temporal alignment or adjusting resolution.”) rather than data that should be summed/rated over time. +### Ingesting deltas alternatives #### Treat as “mini-cumulative” Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. This requires CT-per-sample to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. -Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. rate() assumes samples in a range are equally spaced to figure out how much to extrapolate, which is less likely to be true for delta samples. +Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. rate() assumes samples in a range are equally spaced to figure out how much to extrapolate, which is less likely to be true for delta samples. TODO: depends on delta type This also does not work for samples missing StartTimeUnixNano. @@ -241,12 +266,6 @@ This also does not work for samples missing StartTimeUnixNano. ### Distinguishing between delta and cumulative metrics alternatives -#### New `__temporality__` label - -A new `__temporality__` label could be added instead. - -However, not all metric types should have a temporality (e.g. gauge). Having `delta_` as part of the type label enforces that only specific metric types can have temporality. Otherwise, additional label error checking would need to be done to make sure `__temporality__` is only added to specific types. - #### Add delta `__type__` label values Instead of a new `__temporality__` label, extend `__type__` from the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files) with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) @@ -333,6 +352,11 @@ This may be an unnecessary step, especially if the delta functionality is behind To work out the delta for all the cumulative native histograms in an range, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the range, all of the buckets in all of the samples need to be summed, which is less efficient at query time. +## Risks +Experimental +what if people use? +more risky, why it's not the default + ## Implementation Plan ### Milestone 1: Primitive support for delta ingestion From 2ddc4448e21464f4a1f712b35d8a3449ddeebc69 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Mon, 7 Jul 2025 21:59:50 +0100 Subject: [PATCH 13/34] Flesh out ingestion options and exposing deltas Signed-off-by: Fiona Liao --- ...25-03-25_otel-delta-temporality-support.md | 93 ++++++++++++++----- 1 file changed, 68 insertions(+), 25 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 0cc6c94..a61cb91 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -19,7 +19,7 @@ * [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q) * [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) -This design document proposes adding experimental support for OTEL delta temporality metrics in Prometheus, allowing them be ingested and stored directly. +This design document proposes adding experimental support for OTEL delta temporality metrics in Prometheus, allowing them be ingested, stored and queried directly. ## Why @@ -110,56 +110,97 @@ Delta histograms will use native histogram chunks with the `GaugeType` counter r ### Delta metric type -It is useful to be able to distinguish between delta and cumulative metrics. This would allow users to understand out what the raw data represents and what functions are appropiate to use. Additionally, this could allow the query engine or UIs displaying Prometheus data apply different behaviour depending on the metric type to provide meaningful output. +It is useful to be able to distinguish between delta and cumulative metrics. This would allow users to understand what the raw data represents and what functions are appropiate to use. Additionally, this could allow the query engine or UIs displaying Prometheus data apply different behaviour depending on the metric type to provide meaningful output. -There are two options we are considering for how to type deltas. We propose to add both of these options as feature flags for ingesting deltas. +As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. + +There are two options we are considering for how to type deltas. We propose to add both of these options as feature flags for ingesting deltas: 1. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges. -2. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas as a new delta "type", using a new `__temporality__` label to explicitly mark metrics as delta. +2. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series. -In the documentation, we would highlight that the gauge option is more stable, since it's a pre-exisiting type and has been used for delta-like use cases in Prometheus already, while the temporality label option is very experimental and dependent on other experimental features. +We would like to initially offer two options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta-like use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but it has the potential to offer a better user experience in the long run as it allows more precise differenciation between metric types. Below we explore the pros and cons of each option in more detail. #### Treat as gauge -Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. +Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. + +When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. -Pros +**Pros** * Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. * Prometheus already implicitly uses gauges to represent deltas. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. +* Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. + +**Cons** +* One problem with treating deltas as gauges is that gauge means different things in Prometheus and OTEL - in Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge, rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. +* Gauges are usually aggregated in time by averaging or taking the last value, while deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics (e.g. the [Grafana Explore Metrics](https://github.com/grafana/grafana/blob/main/docs/sources/explore/_index.md) feature). +* The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. + +#### Introduce `__temporality__` label + +This option extends the metadata labels proposal which introduces the `__type__` and `__unit__` labels (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. + +`--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on the that feature. + +When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`), and the `__temporality__="delta"` label will be added. As mentioned in the Chunks section, `GaugeType` should still be the counter reset hint/header. + +Cumulative metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. + +**Pros** +* Clear distinction between delta metrics and gauge metrics. As discussed in the cons for treating deltas as gauges, it could be confusing treating gauge and delta as the same type. +* Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by using the `__type__` and `__temporality__` labels in Prometheus. +* When instrumenting with the OTEL SDK, you have to explicitly define the type but not temporality. Furthermore, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it can be difficult for users to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See the Querying Deltas section for more discussion.) + +**Cons** +* Dependent the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usages for refinement. +* Introduces additional complexity to the Prometheus data model. +* Systems or scripts that handle Prometheus metrics may be unware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative. This could result in calculation errors that are hard to notice. + +### Metric names -Cons -This could be confusing as gauges are usually used for sampled data (for example, in OTEL: "Gauges do not provide an aggregation semantic, instead “last sample value” is used when performing operations like temporal alignment or adjusting resolution.”) rather than data that should be summed/rated over time. +Currently, OTEL metric names are normalised when translated to Prometheus by default ([code](https://github.com/prometheus/otlptranslator/blob/94f535e0c5880f8902ab8c7f13e572cfdcf2f18e/metric_namer.go#L157)). As part of this normalisation, suffixes can be added in some cases. For example, OTEL metrics converted into Prometheus counters (i.e. monotonic cumulative sums in OTEL) have the `__total` suffix added to the metric name, while gauges do not. -TODO: cover temporality label +For consistency, any OTEL deltas ingested as Prometheus counters (when `--enable-feature=otlp-native-delta-ingestion` is configured) should have their metric names normalised in the same way as existing OTEL metrics converted into cumulative counters, including adding the `_total` suffix. Conversely, any deltas ingested as gauges (using `--enable-feature=otlp-delta-as-gauge-ingestion`) will follow the standard naming behavior for Prometheus gauges (so the `_total` suffix will not be added). -### Remote write -Not a goal but sort of supported with temporality label option -ingestion and forwarding - forwarding will contain labels -otel is translated into remote write request, that'll be propagated -Remote write support is a non-goal for this proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added by remote write. +### Monoticity -TODO: label setting depends on the type and unit metadata proposal, may need to be updated +OTEL sums have a monoticity property, which indicates if the sum can only increase or if it can increase and decrease. Monotonic cumulative sums are mapped to Prometheus counters. Non-monotonic cumulative sums are mapped to Prometheus gauges, since Prometheus does not support counters that can decrease. This is because any drop in a Prometheus counter is assumed to be a counter reset. -There is currently no equivalent to StartTimeUnixNano per sample in remote write. However, the initial delta implementation drops that field anyway. -TODO: say remote write will be supported via adding a temporality label +It is not necessary to detect counter resets for delta metrics - to get the increase over an interval, you can just sum the values over that interval. Therefore, for the `--enable-feature=otlp-native-delta-ingestion` option, where OTEL deltas are converted into Prometheus counters (with `__temporality__` label), non-monotonic delta sums will also be converted in the same way (with `__type__="counter"` and `__temporality__="delta"`). + +Downsides include not being to convert delta counters in Prometheus into their cumulative counterparts (e.g. for any possible future querying extensions for deltas). Also, as the monoticity information is lost, if the metrics are later exported back into the OTEL format, all deltas will have to be assumed to be non-monotonic. + +However, the alternative of mapping non-monotonic delta counters to gauges would be problematic, as it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Different functions would be needed to get the same results for the non-monotonic counters of differerent temporalities. Another alternative is to continue to reject non-monotonic delta counters, but this could prevent the ingestion of StatsD counters - [the StatsD receiver sets counters as non monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). + +A possible future enhancement is to add an `__monotonicity__` label along with `__temporality__` for counters. Additionally, if there were a reliable way to have CreatedAt timestamp for all cumulative counters, we could consider supporting non-monotonic cumulative counters as well, as at that point the CreatedAt timestamp could be used for working out counter resets instead of decreases in counter value. This may not be feasible in all cases though. ### Scraping No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. -Delta metrics will be filtered out from metrics being federated. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above. -TODO: not always the case - instead of filtering update -metadata counter + __temporality__ -> can miss the temporality +For federation, if the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has the conversion issues detailed above. + +We will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the `__temporality__="delta"` label is set. If deltas are converted to gauges, there would not be a way to distinguish deltas from regular gauges so we cannot provide a scrape config. + +### Remote write ingestion -### TODO: non-monotonic sums +Remote write support is a non-goal for this initial delta proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added manually added to metrics before being sent by remote write. -### TODO: +### Prometheus metric metadata -### Prometheus OTEL receiver -to be able to convert back to delta +Prometheus has metric metadata as part of its metric model, which include the type of a metric. For this initial proposal, this will not be modified. Temporality will not be added as an additional metadata field, and will only be able to be set via the `__temporality__` label on a series. + +### Prometheus OTEL receivers + +Once deltas are ingested into Prometheus, they can be converted back into OTEL metrics by the prometheusreceiver (scrape) and [prometheusremotewritereceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusremotewritereceiver) (push). + +The prometheusreceiver has the same issue described in Scraping regarding possibly misaligned scrape vs delta ingestion intervals. + +If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we get more certain that the `__temporality__` label is the right way to go, we should update the receiver to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. ### Querying deltas @@ -356,6 +397,8 @@ To work out the delta for all the cumulative native histograms in an range, the Experimental what if people use? more risky, why it's not the default +sum_over_time should always work + ## Implementation Plan From 1bfd973afe0751583c70e16532f3513ef09bea96 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 11 Jul 2025 19:19:14 +0100 Subject: [PATCH 14/34] update querying Signed-off-by: Fiona Liao --- .../cumulative-smoothed.png | Bin 0 -> 97937 bytes ...25-03-25_otel-delta-temporality-support.md | 203 ++++++++++-------- 2 files changed, 110 insertions(+), 93 deletions(-) create mode 100644 assets/2025-03-25_otel-delta-temporality-support/cumulative-smoothed.png diff --git a/assets/2025-03-25_otel-delta-temporality-support/cumulative-smoothed.png b/assets/2025-03-25_otel-delta-temporality-support/cumulative-smoothed.png new file mode 100644 index 0000000000000000000000000000000000000000..d1a61d2cf4c589e78a6a937aff3a40cbe4745744 GIT binary patch literal 97937 zcmeEu_gholwl%$|fEA>J0!mk^bPyFO5v5A+EukYIy{UlGq!X%0hY)&)fFM19v=Dl4 zfzX7|`8N8Vd+)i=x&OiU3r`-hcXrlVbIvi#7zus-N`c}E-4#4MJPO5^&(!enh@J58 z2=d5Cfd4!WZqUcWW5H8=Ca3Y}o@{ik<;|FL`~CYyh; zgEX=EKOO+woc13-_W$4fpHJlfZ`i+^%l~O%k9l6}@LBl1iuznRqqv0l@@ymSUZ~B@ zgjl{EF#|S_=B4NX@at<1QQ7w|E&rL4KZ|`(#HE5^OR%F=Cies zOFtVpBqEy^Y4Ylexbg3|{a`cJlQFn5O-ntw>a-K>lFC~HSq=YlOWx%z1rtci|9uO` zYaLdPexX}lI!&0{q4_t{o9fv%q+iaLk4s4N>phyzR_znczDM3=qNnRaso(V5fYV4! zN9~|i=cXG*J6+iXD^50cRaB{~h?Jc#J3ouEHkb0g6DI-L@C#s1?1pPd{(VpI@I?d4 zj`$UKP&BotLire{Xy5q<%T`W&GSzlaZgv@c?kYbTCFiX-(Vp826ocj4t!7!K`?ou$ zf6G>HaJz-HJrM3}6j=mmV&AeJ@w>yPP1grb*ewtebi46rdV)6(fbCqjC%^ji-{Vcl z^MF(G#%$+_==E&KfByr4SW_moX>-PhS{DFp2^QPrCeh&s`R z8wqxo^Xbx;pP0;%wPoVTO}=e_6w&@!PyikgwF6HqbAo+BW|@+h$JSJq`j-m4;!NcX zfMyx-Hf;WNDeH z4Mgi*pGE1#5?_5AU7387k&dW3d4;RgE~+wKzD5nlQrz}HsjqU0H6`HNb=l^0z>)o) zdPiIyt-tKGD-Gg4-zP;tKVAc&?8YQ-1wve;UBCfn|b*~kQun_`Wb=lUghe- zEq!XJuv>VWmHV~#JG$JW7jaCc{HGmGAbq6>^KczLC!2a=$rlx_@2KWEa0rb-N-vS| zc-lD3uV{K=2MX%P0s402cAs7@CkAhxLV7wuYclJ8w(9uDbUI zujvOo>vXkIwd_J$+)3Xru7Ojm*EfLuSr=0qjP!W`jlWw7>0+MIRytJhiRFxbbWoY% zqy7Ty-?L{Ks|yR+-b=Y7F6{f_@`I5c5_z>(WuEABogcR^Ko@;!6vyZIiG;N(-njvL z;`Zd3z^fAx-FTW2OsAKC^2QP|6GTPkoKRi+FgB!kiYp{p*sZclXzTVb*GIGj;W~gD zm^GbjFkYHXM$47A#I&yE7-lKid3~x5`x275$-7N+!xpxekR?KVWtxeribAk}CSBSz zlWm{r&akDDZ7Z02sa>FTU&p~`*K$cE8oDGGd|bw1ha)!0vAe?eY$nEd@!0ltfw z>s>DuL+bY1#{;*+D<#@JfyffDef4&eSCmH4NxtVjm<9bzmR{7YiEO}$vjHcL#Hf)N zni+ogHF_m+!gYVFzR+~DPZDvuXXM}EBxmdvdE$J|)1#}P~ihV5Z#d*>`!Ys?Nm z;eBNVGL8Tej@O6h-_%ME>eme{jH*T$UXC_s)mPY^YJNsEKDA9=-e{0!)e%s=n06}B zdO5kJB)1DzTBaQWyVwNP=h)6c)OPoHg-NHAY?n8bd*|GjYss{S==$$)OH+Gwxs__) z4jto$NNJ89ePM6Xf$JLeer2bQtmYRI|vVxGtePNa^vk zRn?U#692mO1_6;1{mHH7^FKgCm*rp0vY?>-JEq0JnQk%del4f>{d}bUW!UWxT?WtrCIM-B;R?W;HpmX9>wCVZINg>m1!QnTnt>2prlO^SK(>YA`A_Jt3T z?BDi{6Vem0j{y*p@C+Z_V6O!3@G(W-AdAXPT)Xa#K!2Qj*8+#gpl zuCQH`P_qH|gBzzmPzEcRPMEkFKDC(ERp)lg(O51P(dqKwcr?{giuy2hAVfP}KjLvZ zPGu}wne6wNH0KWOUMo*jWVdEYI!8PsD-)7<;)-#liF`EvOI9A^Q52am1p zZ{Bqsa%qeG=@ZHkGlkL@t*oPLFw_@^)|$u-0w50LeUV*?h~wBB_CijW#{?vLj+L88 zr1UhBz~h+)ZW8@8N?9S}@q+At>Xk={IV4aO-GTd=7A#!*vITg0K%8E9&5J9V9u!;1 zS`5aVMVgkqRHr(XQCxRU;p{gSaL1I!@p~D4Ml>`N@m$r{x{)wFXf9&(po_5h9YYnh z%8>1b>B8%yt$SNImfSCFTOI*yTdp3~R1XBvXx%`0Wt#U4o|e^$J2+IL3TB6++d;q+ zq-^W%Z=NW?h{XvEuoN4nORm096u^J4s6z26w26%6^;P`9no8M)n}B@+aXr)E(t4w& zK}%nUygCmuoptI;UsBsYjLN>a-F3W~t)}~ilQVQDI{G$0Z5MfT8UsI4AjW*yx4x(O z)s4Q-q0YwM-%m+OB`>X=fZ&dw&;5_giL-I(IcT^pHOsdVX^vKxknhDX1%KoW3w3%u zsgvNVvvG!ygWA|WkI5D59W*05WP`3WVM+1)3!~d;@&2gFCpd}F0+RVLyuE|nBC&bl zPW15plSpo^J(Ob0hiNAKH;>JppwwYd1hpO|n*CtjFWYhOO~i&xIt1T{`c#J@Fy>}n59D%;xz`tPrjJRFpEt?U~ zBOL)&@XT~c^>6!_8=1dM{qB03T`-jq0(+8COmHsXY(qRyBc5kzdZ&wRrO< z$fL`)U`!S?r%5*C0rL)g!t6H$*z-H`qsNxsatgLIN3XeS8PUx&M4E>it~%tE_!=Iz z+)&#=q44{^;W#(9X&MetVbhuXy*`t1b9CFy_2TWxy_3Bz%ZJyn6XymGck+>JeBUdV zwmt6}w&@~t=FoD@j{W@zY9a*MJxjDl><)Vi%Dhi|T zu(vq4(KeB^_x*&>4WuOX2FLgPWqE$~7Zw!7a=Ou96z4l`ieK2dq5e0G&lnyYUOYU0aZS&R1FsAodO zoq89>_IVDR_&maiep@nJcC5QyYIolH{gu( zgF2HpN-YoJd7djXLQ=}dl}DdLvb~SkIHBGJ{byjE;0pQl14#6R5*2``4RuECUFpiE zD>r=M-w6rGK3pc_)(o`7f0J^XK-ec$ zF=4ByA6xJDMwffKRQug)r?;M0Zai|bH4>|EFV{6>Oiq}iPsW97ULx;T>Hz@)3qMH* zLm5$>lBzOY)pXZ$Qw3giJ7W0$EdVSd{7}ZW2Nuav?fZ8mhY9P1{Iy*Nl14JSml4(i zLQ+AkW(VS?oVkcANY>LhV5`04UPs^Xs* z?)V;7V52S9-H#!f!ZK(33ob-Y*)z~3GW-*)G0V2`{TE|&j?MdDAfzuZ4kT0)tH5-| zlFK_Y@|7y^1?%uhZc00T6f>H2!)3%R(B0DB`4F5sV|G#<1);dM{#Pjny)E z>6$7V>GgwwxFKYSA4yls5t1;B;Zd|62XTQYw2;Qbc!`bgzdVRuOLhU}a~2w}M?jYu z+D6is^!2b#AT)D-(jakyPM}rVN$__{I6_O(H7xe*=(DS+%WKW`b6jFw!E#BN0cKY; zZDa*2r!R%ay|O+(eob=)!d1;5bE@rG7xuknV%X$j|8BkaPO_SiMQl&M@T2u7!^r&8 z=jY0)*+4!4L1d{Ycn~>cKdLBiv@XBx`z~edJZY&e?QAQ0m=#xd+A9{4JOIw%jG5Kx zK^fM(*-dj$_em|*ys0LYR-H9%6&gzeW!Zhe=VESD)&j}@R1m==H)%rk1WYmnz=Hsg zZroq!r7$f%@`nQ&-_L!mvtCh-0Bg4!q)hPGPe{a>Y}_RIH5Iy+rz97u-8E0ur$?A*SG2 z`6?j-IYe;m%$6wjbPD&88P(I6UgS(Ws)u*lh?WRyEZ17nBpm{Zwrsy(6xVDi_^ z^r6~1hnBWRWFbx%u?1c(uq+X;{;Zjv%raZbv;d*D8~+6-zGf)Chme4bV!MC8L%c5c zQ2#W|&FN*d`IVKfldXf$Y06L9vWC&Pk|$SdWTh}UGoyFi*zE5Ykw|w z`Sc5BOBf29#VKPKEc;*KBo|@~!C&lQu~2xg+VfWTl`W zqq8Hp%45>Ec0$5$f)g`-yDGekdV*Q}F#uu+_}Zk`U?3D#<)#1jJbwcQ8vUX!8*AG3?ai zo*T`>88y{K!@8vv%^-a8m&71}S72=3#x_<%u} zMcphW;WOo*g4CYCEygYxXA2Tzqwv)wb|MVmK6U({QFNYJ2WhvN5vRQGVt+jD*Md1S z2o|I*M{6&V7?8+O*>In3TPJwtVlnTCTtK4*3;P~lg2oy)m3y9ATROS!Me8txC>xyI z_H#-D(&|RfyDP1ThVax2wFkG%g@b!H!8_^fHu!k{rH4T`?b~WhOhaGtuvctWWa8U1j~0p6wkluG z{fk!;rVVUcV(lw+SEt8SZRsHg%e-I2W|2cT@QDI%vV_pzrb)da%R)y~6j&3wtdUn4 zdCi4KmCBXQM8FlOHqKx=fN6(`n#f*j&YI$&z{F{n zQv8&Qn#zG`fso?N5tqK*cp2zVq~>zn%pn*#3K|i_4U=^05zlOGjl( zkuqv;|A$@K*sz0v$+=XQxxkhkfiI!N=C$*W;_nMwP2rF62aYgT2n=(?L5;q$jaEz4 zacJ}24?lMo1~>8HQ+gd;+WYQ@o{;0`pUw@)swlA0_xn?adL#?GS`2kD=N383Tiv`r z%XBXi!@v{;;N}{SueuChIr${Bltk>hL^Ie1+`F*hBD=o>J$u&%81wK zH2yY7YUYjJ6Q8!^obw`&eVGqPshD9d%!p52*-ZM^c?Wpjd8t>pDeiHMXDaJj$9oya zKA+RiBkC%6XNgc|JtCVaWfAkV$Do_zl&-V|w$tirr&F5ox;seE&!hTol&A4P%56?- zq_-EQ6YJ#2T(o^ApKZG7J|Z(pxc!?&GSaeItPH;Yg#?%3PUi4Ho^@dLvK3jU*Z1j0 zh!1`tOEDJQT0_W&Xya&QhAwm}QBR-HL`ZmALux080xRCh zXMiB@C@&CVWu`7~_SvDJJe`l&&Cj9<-NDP1jt-mAbDN9B3kH5Vl3!(kuDCOcx#On_YjMWiI;)e%Djk6iWcD}Y z|5S`0-m>7{p!jED-7G``7Hj-C#jY?YYWSA!24~U=5?3?qBaN&W4(q8CLH6<{xh2Q) z3zAwNJuHs`x~ELEKbTy!V_#x#*#HSp?I;7D)mJ1Y#i?M4tgO8_AETvT-0xMDtq?x8 zD8(OJao=!PJaVPQwwQL*)A8+675fP4!M^wqk<-i-$0@fMP-DXVF`bd3Zua-)&ALSU z%D#1qN><6O`;*C>Mqlk176f$by)#Lx?=;jY?DyZ4h>Qx0k-^nsGyD5vciFcNrwY9q zEqz`^*D*0?kWT@<%H+$T0|n9b5HWlsxyhKcG=4$tP0x{s&8Y^Cvh*I!%FVV*%cf#h zr%x4@?MgN`zu=;RJVhuD%iuJ2F?+rGeV`!SLT2Z)__MG47tu}}geaqvZu1=MuhEm1 zw@t-<7?j^)Je3v^!zrEFwJnKKh%0R{$irechFl=x(f%hvwO*S#d?94B)N$zyq4Z%7 zxKo|9GWJ2gvsBSN|9ttur4~-eqGAl=COp5x$*VGIZW+B=>-Fg!_$!@8$&80Lj%TO} zq|Sgp`{txN-l2EzcX~hAaKEtbRk@1rgwUj20q5Q~Baz{pthUdj!A}Cm1B3vZmc}vb zSLI%7aL^jqQ@yM?Y_3&4FLz)O`F+{ZH+ScSe zA2iR>66S{wc#$~CCpQ-CdwoG5MuBMW22U_9X1bCCA2|W{k-FF*iXQi{Hz|rHC~sJJ zMB_AGA>-AZRcu1W6C%oVBkw2L26kh4OQ!-jf{vs@m2oO*+fQbDVGPgCxux{GUq31H zs+@e9vNdDz4)Huj^#*V5>}2$b0h>)QVs=+>Q@iQ3bv<2&c>jwuJ?E10%-)#ZacB$( zZ~sZp#Dcfc{2)~Mz(UVwx+8iE{&lccRRi~?I_#^Ar@p$-kw*En;)sl@#BW-~>7L{3 zO&J%?j`#%OZ-ENre$$aHW5C-x1G581aGu?_O_s2A$e|V}=$ofM8h=?B;km-H7+|$AssZHN~YN~p$y$US88y%sJ1%4k3!T>n(Lv*?7#c1w~Ce2i#lLG>EN|vbcZ&<<$n^AOc(Vl~PtJ|udpG!@C zOQG5E6UwN)k@#c*3I$2=eh8knpeEG@`WfZgh{|#?GNEg)>g_a*s~Y^+TD~{Uhxo+R z&OMoDNS&nWnoj=Jsa%p?VZ35|la-s~5S7|U){CS~xt;B`silpxqV%KJwKfMZ%VJd3 znu+VN`-7-xDlO6|S2Qo^U^9C2iAG<64aKI}adngws6@M`xM)(1EqlpL%<5vwLPq_; zSnc!g4t55$l$SLd%P9dAokts?$?G~s$kFWKHE;{VjX7SgbLnX^t#8Y&Ie)dp)JDOU zu`ZCusrUMH|2a^+-&I_9%S7o`1O3!x&9lNr5|<&>L+h1&L#WsGFjvW{Z^+xHg#poG z>(v>X-UhQwDsF3#!18?gUVQrq|KELKV1=rJu2`73M(>Wif^6gRWKeXvL)@T_Flnj{ z4N@E=GfngunTQ2JCf0>5uBset6}NU59KXMcN8{&L8H@BO^L$w74{2$`XR>&7bAl=% zYzVTxUkCr$I1ML3E~soz>T##;`&n-0gtZr2>TtD4s_W8^8)~`a87=3DMGthZIo3x_ zYm&%k>;Scvog@yLdT+p(==`w0dFS)H43JS|Q$K_E9hLC0$QApqJ=1v}yj%mdMaPW} z0t9EokN3r4-#wSFrs0p9cg*1S!$3-P1ss9`2f}0 z%HaCSm8wtJL>R9y__+=GN@30(#mS6=p&-ZpEKFO)Eo<}@3@q`Piy^?fum~gOzR3-{ z@7!Ia3R_&autXtwr)NI;1Q=o?`59i|?Vdp^3N1Yh`4H#gY$+410J} zc>1|7OlRV28-dExg|CVa91NqS7z2LSRUL1`Fwk12e@+|du~Ne@x+6Cw7+Gm^RVX}7 zd&J8;6XW#Ua=l)w3GxZc(phsSIYq{vJ8$)2mn*$^38RM3Zo?BGmiE7fuOh!e)b&pi zAuh$gu4!ZT7$(na*H!+-1?aiUMpdIWNSRNrG%nH~6DX?UA((4BGfc=NbQ^73&`5%G~G$t-V#w+Sh9eeB-s zaVI(AErSZbhk9U{E$#cuw6k591ot8E!n7X+@o+C4QVJ0(qlUikVS#(%QUaEdP?jzq z1+PjTy+df%b@yX^!f{lhfZMAUFl-BR>YXQlpy@gKobZWEp^Spp;-%SC=07a=-#fx{3;Xh?g&cu&QTM&E_s3u?{u%Rsv znPWPfWr*}uhtsn(s#wmq7@<3pz#Xd{iS+7~*}tWGCHMUsVKA>5A8a~12Kmi%8cM>R{s=t} zl#5ykHi30jtrSC&0`aQj_1ThwT)){LROIZfCeVm?KREEi_i`h>wLa1Tj=-O zv0<-w&TQgpj{zS=mSnqExu^%S*_}xx{Z~K8@;GgI%Pvos%}2h{vi%Q!kiliq>9?}p zuM>~N5IH->FMaN9d0o4ZQubuZa1LJAE)@dNR2@M@e32G=)2;12KgeD>^d{Q(HzcRR zAIC2x>{V^s1Ytz?3g#+5O=gWBT0%@3aJd)d9rY4jsF6VBE29OnOpzL;XJ*q(&Xl(6 z-y9Q?gG`Y!@}pcm?6^`Wf3R>(qWSuqxK+(@YERLFlOLK)0l((Aiet+>R8JZw^++k` ztkMJ(=G9<+sys$V`L})a)zwA5E#VFM756v-h$F&&O}gM*2N+VYxgMHHqcjdq%9jVK zdJ0cg&xTG`VL;no4H$E5?v^q^b4XLrK+B*e!E-jvMHzizA2a8z3bU)Zj!%?wu9`a~ zBzrmMd3&=Y$@-1gt(0BVE5ZFY>11Wn1=Y4=1+|o`zuKHuzmjAYjgzaMnRt|pEEXR3 zWV~wX;o=cwC;n|Lh|7`dO7PX)+V_9VduNVU!ab3bd1pG?>o{vL><8;b>ZTxGK4HZp z(od-thOII=Mpu9Nr;?;rHU!0sSjWZiKy3<0n+;B(Y2KLehB&tY64k#=k z`1?MyLn9IVj~j#=)3uqWr{6fYHm*6`g9>jL);%MH>oSX?JMaUB;Emhpp$l5ohY0gk=9?Q~kDli*RJ!J7BrRb! zr7ii7=k+;_gnpGN00Y=ihv|tMC1HX|Rmbao7A>9~tojG{`U%~9Mx-c2I`dl;D4u>Q zLvw#--wK7!%q>Khvz>HHI#HQ>CaVjY3%)uT9p0^!_8#S8!8)7)^?_mmxiiN;olTu* zzlN)%wl4pB?Tm>x#Vrif%Y(gVP7MB4T|_}K0_>PF)(B zUfzd07s<4Iup(tffV8(@X+VBM|3oJw*;;;)-#A5YixH5$zY-PNqaJ`q@R)PpC}&N& z=Zdx4Os~Yj4(+Ma%pr+-u`Hz4^J>WXjLnn^QU#@MklHaIs}VV2`8J;Tt_9dm`kk(O z`sj4v8(qpD6PUHS{`_UUav9W}?!1H|$kO>)=cDv&HUhCI%weP3EguSl9`>4QKM?2* z1rg+IkhE7$x8HY#wsUB_)gjCk*`^8iVJ_)2+^6wA(4x^HMDA@BrbBDi=j{x%L=DqJ z=_A%x)i~}A`_WG5ujtITtQ!uyPQz)S@<%Gv#hOF>w9*Zk>DL|!LmP!1bXLdxPFJS| zet0|0!aPoPTWc>$bIv!vtn3$KM5Kx%V;wqa`uzX&5=`zqJpL`DE^$EE!JsJhN|@9# zNxlXA>-VG5OjXqxv4{i#a?i0iCQr?gKv*bb;)(htKg98MA4;UGxA27Suo4W6H8>8f zC@J@ypFsD7vDwsMIjcqzfO9a^VgohNJ>85eM*k4+E*`wXc{0uBIh%-x#sh}bS?0&7 zrV~$k7#wriiFlyoArD-|)zm(nDg5>?lhxbNHUlwE?L=UKl+t=saCA>C#;&OUnQ6j; z3JXent?leyhv4=>-!*VYjaS4@ZPt9gS=L6s_9@@Q8>aapBam@T!c5aUz^|Kua*<^D z89$IU3*bw+8baDUApz!1;zot&M$#mnw+>5J(qyU=tXYP%ldUKx#eq&LZ@qagM|boE zc5CnWp9T_LNd<``LI8hq%5CKHi0v~9qIyx@rCC^U9#e>GXKOLswARRP#VK`ZFo>}5 z+d51rkOU!2hMhbmmK@O|Obx1E)F6%AFDdTGIcMLxSWY_oW|rQh%KeKFyIu}5DwS_< zD0{oBq%u{92PIj9OJoKKiQ=!m7A<7?ZW)8{Qk=bLnRNi^J~j1o@rRD!`+Jt8#y?v9 z)s6FeCTX%hex9CCSLMkv$Gw8vsk;WuT2~^B8hnf87GT3EuXX4tZrDpdOSqwGj0Vk^uuf))rvWC<948X z_ouxI9R4ciNz@Tt=gFhv%% z!^kWaU;pf=zFefMa*Jm#aR$|Z4UVcc@?QG3xio4oxd$*nvQMt~0VzQ@BCuO2 ztgt0Ib^8pihs+x(T4zA0_Uue!s-h{#F_K)Rxvu4P+{Mt1%CEi9b(>D3XH>V-k`fx; zQFTCq(=~mz#GA_X1td%OSp$VbKRA>?GG+db_$%)W;F)xZCMf=_ODABu*W|&h&^VOE zlzW6|HxCu39MoNVw+*$-h7RI+`X@0FP>)(F;VnvQ?Pb@Fo+6vxslUy&%?r{lI_n!= zmFhF6N1aYH&txS7quiwX@^5>@27tZ%ec4HM=neq@0YBi^^7Sk@bsaSl4n0s;_jAig zKQ6~A|8{#AXQ>FL(D$=b_D#ChfV#bhGA%V6e@?n7nbo7Wr>$PGE{?QMahhO>G)}J$ zYBHoB(x%+Pc)SectBiThI9T8jAwnC)f70pyZYJ1L5R!D2JLn^B>w#y9w&qnf!m;$) zla&CHi6F*mzB6IL<9SrVdxo(kNUtDCbu@h>Frh=57`fVM3G3moiR|eHjoPyED|fUz zwbc2(my0#`uRDtQU{7Z3d^%@8Ts`jm>XnG%I1`d0+nn4P&AwZ}=d+h7wK!`CYK#~@ z)4~XF&>b2a&#*?vGi*7~Rvt@EHdX50reXcVTzRb!(0%I4!R-#shaD$>Kew|e77J=V zy_b79oR8c`@V)|{ymm$a?bgVH`Cm8#jCpvO^KZ)x@bQ@C6yj7M{< z`1;#EkYK#9iw6a+vk3NRuC0?&d@D#MxJ0?F#FTvOzKH^3W&BTff&y9u{a z*T`<4fX~6>o+me75Y8{W_yliwQR8zq%_jbr2H`LJnxHmvlqjsLjpB1eA7F;WfJJ_v zZ%P*pghf++5di%Cz~+W9*A>)PVT!4|ax&NDCZB!gLh zRQ1!la>{1ULK}=^du5K>dN@bP3QUJMup`lb{EQmE5@lr@t0(=|L@XB3nl;JK^=OHZ z@^Ho>6Wxpjp$ai>_F8fdF#6M$C9m#Qqf>UH!;$kB8#Ph7|hexlM z3X4%s$RPy!rcgM$k@h_|a_8WuSErm*qt*6tGHPaJ?qFY3CjBo@$zRrypmw2mOTC^_ zdngD|2IsnFkLP*$-u^>?XT`LWM2AlV=HKph$1KXECu!GDj`Q9Z@6rseqyifwS4Vmc zaUv5$nd2h+XG$lzeztL6jP`n;{jS23DpICWEC-WHKT#0sT!F1Yw$lwO6Z&VzKG@JN z9#Co5T{DRzFJ$!9uQRCN01}Sg9=O9@7CN?l)MZJ{&{@@n`Q|vJVld2CpIv*ce1@>x zVzcUILOSEG>e>cz%N2aB3utd|D8e?Dsu$l9f;6%0W2e(iBy6S>3J`;-OHMYJGI!FQ zZL$ugW6qgQJYfyeK4!8l`mlhamUU@6fk55$8SEui8 zbNudaw3Go~>3S~himr64QT^WcxXZ!ascYf+DNvy;U3P;GP}dxhMD7mf>#s>AYZD^o z4q%w$nm-jY&{WeW-c9QjaMgD|vp1)l?XK-3xpA2Z3j!8rwyah9n&`SC_51T+SLYe? zeikeR2WC&ax=7W}St~%bu^si+OUWL+( z(Cluw`^Lh~uWwZSaX;w|1ev|#lE>a?uSM+yS-NYKq%b`jMlXP zofIe==xcFmBN{cN1&#cP_S_ZH6?h#?M_x!HWPJ^nY1?2RpABQION~#E&Q+g(D8S7K z+4HNLS!Ef^ycsDFrRTkKV4=%a6E%=k+^1Mmy&SLpQr)!~qvdHHt%b~Q>9omN&SN?>zmw=?si7)nvlY*$%md4N$8cpC zwl)Vo)@?-O2)~ekwsucWn7fQ6pGa5qm#UufxmY8uq2 z9UjI$Uyk(plsm#QO)zJLIL!4%v@$aj4HD8F*SRNjNeD`*8l+j2^-g$n3Cz@d*Qlr}^YU7)zzTr< zP%3ZF?mb7mc5Y+a%%>)Rkd$|F}UHG{Hj=LnY_0*-*}pcoTDE0E{%4iR3)ftL38VB>8N(l1`s|V% z1p*ow6;n4xfvg9g+`0tA7el;0U9s3~rM-1$;4aXy6fD8;P7~(os+Ysb&5K1_cwy-f zP27mbkA65>85ms)u#?1hE4YYy6Gd$)OuV^u}Oc$JNQib=HbhfceFiRUv@3k zE$6{s1a9?+-09lBNWK-T{V3IHvMKT%IBnf2H2%jx^j$0ncO1^S@H+Vz-crP^@3Ze; z+_;6NZ9o;RsjH9^P?&r5Sh^VnlejRa9w+n*5Q<}ePaJ$L(zp^K;E&x}mVimi&fW1gx@p!J5J zo#og`g|eZW*)s+RT_(3>7)M2Q*jr?`_Jw1m16G=U&?& zwU=(pSGs{oc`N~OO-nf(W{y9T7NBhksvl->T;>irxg5*@0~&ldh$?F^C_g26RPAi} zQzLK3wk7TM2Sf4Wf^e{_qt?hHy2Vkc2kb=dRG7Ny*ZOdD1r<~tmy#^rU=Kc{eAlyj zAe`>|#&II+&hF1ZKi+yNUrQ=6pL8wXgpJMVw8u4ZVf5(MCC^*#@e8Gc4GkJ0nTOTksah*M>z6EJLPX6yL{+DKqZ$vXwoE$(gRbT{ps zaR>pK1ApD7K#kJ^i@)rCv_E|PXMC`G3Waq5RPnfFSjDsctf2PQ)-+%MTX(6X!JV17 z0zUm*mJmd@SO%V%5qj;3Wg5dQYoa8qUXN0rg_e7E+XU(Nuw8qklcbeYuds>MHx%<* zVKp4TqVg&8z_njPa&lP42RST9k>hcd2l-goRZp~3I?rxB{i1tJn}$>)L3p6@P`nM) zao5B!>B0x*${B1jl^oJ^1+J6SQX#y2y#gq&LoDw?j3v?LL57}=67X!oKw62YrfEYs z3Zg(|$6L)ffxeHo2AOPSO2Rxen43ODYwe>{9Fb(z;1x<6ULcKUhg4Db^!K=T4$D+` zxv76g6&w}(EEIMZzExEP5_A*xnmQV&@^kJn-}183r97I=pL&MEPQE}iylm24rQIGa z)lTMj{V{THj=s23kne9k9{#4;u;KY`n(LX5lTO-3qI7C-HXPBi z=cA?=7eiegk#DJJvVH{Wk-z3^9N&RiWroUkA7FH2&M?x;1h-BlkCKk0jXbC<86#7F zwccR+J*s8P7fs0g+pt^C&YtPl8$QVr7^)X6b=S74>wOxS;Qe%96x~}Co);UmnyysS z1(Q}0*5at1DWCpp$|ai7;>K@}0cff2CzFrlvJitQG-P)%#-o^aM03H~;4O>_13wZx zaBuW&QkYOVTKAl(oNuuBaZGPi9xLn*(R7waWR-Sbsp3*_{+X=2)GC&itvu`s6Kmtu z=T&Q()0J8HqhDbexC`Jeg#92BIzvD2QulK7W17RE@C*8bFo1W@3s4*Dwx|l_@$mTx zES8P4@Ns>+oeV{GVjl}^rc)~w?Y$j5nrZV-k~8uwNsh&>*CqD~a42fHC&FikDRbW~ zbz;!(GedB=W}^RejbNP@gFp2u;kQaJVHV7!CDh;1O9tNrY1p`Pw!{(jcB7N!zzr`h zqHq){B{oCk2OFQPCzS|{ZGtkr$_*Yq0@EFCSW#*X8Jj``4egyicFpo$rx0I>(>P=o z%XDz@7>N}Oc{U{9BosxD9fcd@czYNa2Ro)5Y!D_;IElfDG?un6&`nk2I0=nOUL!>9}bEQ$mvV%GQ%& zm()q2D37oC+!7t;Pdy=Y4GkAr#hG&V{6Il(L=&60b8}w7EOo&{@+rOz78%wrENaGq zsyGH1j@(F2+FNgx=-gLuBl9fC6i6Pvik{IM@o{W+ZgG~@E>rzX z3+r=wds`FzTi$f1^~;+U@uba1jZc~>zh0YP_Z8ya@5r7Y)Hm3yn{qd$QTXJ{vMIX9 zzYYf%9^uDuuYkJB1P85F~ldR^-SkJDC;>W3HwiXdo z^U;IsCtUrznl+bY-7gNn#Ky5?7&*Lv45HtTg1kW)Ai`h-0CeUxjGniL{a>Z-skG-> zCh-XgD_%SfFz^Z23##_)RDT*pnyX1sD3|YdwL3c@A5CB2dOV&EOe7^+n0`~|C6Q!H zOg=(>Kj|XW>mITqHT==69ljuoG96J3T4FXpt9n*bHZ{DkTN%kX@{=*T-|vGcu981& zD7G8Aebj9}2j&cvok-$8rPVOjfncW{2w?RVBUoh;n9WJY%u(57?58o4G&w2bBKT3g z`kh0ncT<{Yzs2K6=k&(3=IOS?K(Xf#kd#UQCQW0dBI%wSWc!2{Iaa@N23 z75>C0TD8MDvag5ciZbUuini|8g)T93_$t6nJl1O?xB*Conr9}I%s=KoD5MQiBzHQm z+w&kJuHe{Yaz9^xoH@ekaGsNDR>s@!=kZN}*T@g0<5Mh$DLCtFYSnR;A7mIiXujNB_p0nzC4w(4gO$1_`uJiL&v zKl4X8q)*JgNFI#n{yJ_B5b@bJ3UX6OY+X9o8gCnZ_{t?eiEj1D+-kMC**BHMsyEZg zja^pZCJrv{;IAAV`wpyh%_Az?<042r1VA|79U5dDv5*?x%#%v)@!)X8U<&Kv`AUEDjm1l(oKB&=Bn6Y)H`b)m5EQ4G-a;@H%Y5*F>{%=q=&6$uaN zmNRmLVnunyTAZ4bmRq~mO2OLSEp>{(buyhR6h4` zGbU9c1#W^!YzxrZpUoVOx9BSN-Jhh$DB${q^<2x7v>U;O(zI^@3N=|SisII{%Xp%qqf2rPNGo*oBel%FzZ}6*6A&Z21OZlzA0#C zO~Z5NUzh$PS`1jF1{O4h3m&;N=o>dmlGLjWld9wlPn{ZzW(>iU*Lq?xksB-u*k6;F zQ{@ZJ=pcX~v~vW|EMC@$d53-a?^FBd)wjzXqEmSBM_Z;li;dLoQkH??##({Cc|5?N zEt#AUoPot$cvp=tmFLhZmTjM(nT`kk{HHMgug|+~ciE=cj3{BctcQLO(7VTWH=kJQ z^g-`3KLC8+h{UaT_we*X?s*ita8KXY_kFtM?)XQa`wfIS0Mz}nBY&5fVFEa#-54%m z>T;F7#7%}3pgwxU07Qmj(brkik9Oc2M^wG7`qyh0GW}{-s8WVC`JQBi&xox7m;ZD4 z*Es^Zuauq%DsLPpDcVZ#sKh+txqMGH>(W?tQ;se(z=yXEHzHxqzaU?Hij@d!BU91x zukMy4^N3bDDgb3*6?Lqa{$4ARFk7MU_rnR+o~A-u&-orw5<ITuvjgrzT09uHUOo#zOB$I$olUPK)SZPBr@RQ5W@h@z3T3d z@Z-Mr{pw?@7yGd>2li`l`hWFHfaiWtWZCNCDNUexAC~5ON*XD1|CS=? z`^|Rp&AR)q2UND4s_9ch4l@7hngI)b_9EjS71^aY^RbCOSiP&M?bPe3vpID$Kz6=d z0}ZflXEl<`nv(95f4pYRQqY}UY!S#%G%Gj>Sz{NK#nvtw1}eY$FMG^jCD(`r6tofL znVtsF#R+d^dQKWq*zfbU&mTTq7^Q877JK7BpnxS7}%@l=oGA%!) zV7RhoE6+_&+`si70R&yTQ{JN^EzXk#vGkUn#io@cyzRm!+^rkj^|2}lJteEN{+}6O z6Y_dMn0w#vw&JyU-Y4Z)Ko}8f*K_KR)$aCnR|3S?1nbHVE*WmX*X&Fdy#K$yrpcXw zW-D*7Al!n)mfkNeu=)UQ7-Z>f`9}_PJnn-&rD-VU2joCGOm>k`aG9;Hu=R{G%CxdA zUXHw@j1EcY^FnmLOBLnKsw07nNFlrOZ`x zVjb^%pefO#b2&pf1`zbcjFidq|F-16uZDG5E|#!rwPqj5-?b}`uF8Ni%Bwrx#(6;4 zFS4P)8nwK->a1b0{ON1}-(g?Oeb+ESdHkCy%F}0>uh;JzjNIIQTipKX#J6atdGk=g z78?hc>6EG(AXbe>7d26*`2J9c9^QP`!#7t^@Ky$NWk#ygs>*RBlL1|0)gODhV~{Q2 z9=Fi3gFJbHEv-fKL9~N_Un{6RRSBwCd*H(rnj;yM^t#*EPwO)Rh{4#RQiwjmJS(E> z8t@|ml>qeWLB#nkgU4U|{u{V~<)%P-I9z3Ze$8J;Z$zJ1d+$eH4*K^}YNKUZh`b9a zn|8}8FNK&V(c!E!?*Vp~>BMRMQjKSxZhOAbNy<=_Z{6(%2)ILQWCXTuM=7hk?okKS zER*Kf-4hN|_)PUK%5Q^cHKh#y&QSC?Y~L4E=syR#Wep^buAt~BuDBxzBaW?fk4%B{ z%V~mm?Gtk-i~ZBRrV$zLKU5O$24!YOZkONpP(=pnxG|^1$>L}_Pq~`1BKh-@M=S*4 zuK{$q->_!;SE~PCE6kX?@`Th0@HHWeo4$KvMU9T%7o{8mKh#4>oP>?7`KRXxMe}hw z_m)1YHO=2jNVobcI2rGz_I)3wQIt0CR9nn@GKiGMw2Xhb5N<(I^5H8oKFseIL^UKStQHwoP zGovy!=HrcRIv(J^o~s@?(rC zJ<6?h&ocn7IA8jrzMlQ@194xis#y!p^Yy2Z+`^s7wglt4J#Pv3*#$4S4}SRRAB!zZ zyLj5!GopZ`hUn7c@RB(H*rloJ>`b6@?YsG#Nmbs#J@ZtXQz~gpBkInBZs(aQ|CMZI zf>^fJI86DGQ?)z_ZWfoa1%V!IZ=UDrDT}Uaij4}2$04C3=`;<3y|FBL)fq+A(O2e5x_$0}vNGFT z0of%Rq*nlrufM5(INbusD5yq&;!L3UI3Qj(ulI!e@WeE*8J)x7tp5Tt`~T?r?m()) z|NoNAvPXz+vS$d%x((S`QCy>BWRsBXR!T}{WUGkmtb1)j_DJ^3-h1!gdEHCn{rP?W zQO4`M&UucCnh(GO(2TLI^e9>neDl^9r2m_MPOX16;3j z>4+S#0@KJGCh*hWqhF1p{Ej`pU!1T_Zk~mvWAaY!yi`-{Ovy-O{(cOtIX85XeNjg` z%vel;>5cH!ww&<+lsR{BU!uM4)6!a316DJ9=^^yZ6C3npqy^!}tiKbUQ#1^F@U$d^ z7y#oV;3Ho!ZB)Xbp#bwm8>^DDE9kWk^qPpx_3yQIFI@P!zHp86iG%<2zDtc3xWM=_B-d@s$R5^7 z$oIg+K$C<1*DAPGi=B{AI9c%69kdQt((|l$1Z|+Ha_6{htySQc_EPPXMPPu?EPtxT8BdZz1XRDGuZeErQsseHksy1Un|F0xTO z&8u@#mza%&tk z%o_cbUzmYe8VcaHa+Q9TUfq5aETMG3^l6+pX-b;;(q}xYy0{sMItBE(9vTwvjWYYj zj}$VU)Unywd1)hy!l<5nP50m(KZn`uWX;rAHt&VAzp`W}l=@Zs*+E|s3!|m@jCGO+ePtG{{;uCJgyKZJOYZ)n*(k~JMVddtEsL=ZVCoh=(qMq-bUuylfIoIgg_wH<#56%YF2YlGu zXy3+qQop7!(tRDu$+C*DSaKaWmE{OuH1t$8Nrwq2T7Ov^&=GS&S_yoKri@ zDimFLX|VSqvSgQ5T+X;0PquW^M3Qx8H$o@JK{%xtMoTj zRNa81CD$?O&D05HBa8d{Z&(~FM8(guE2=L5vbDok)S|>r&C~uU%Wn+1-|}pxk5o!g zeD`GVNq05XnU6x?_iCzkH1MYrWmu**16t(_78O^?(RFV7_XMk8g194@+<&R$UxCB?FX6O^D;oj%koO zq-@SG_|NNT!5q3gS)DWLd0uiEY^PJXixm{?TON)~GT(mjvkvzaw-XO-Z=Nw*WY?}h zuIpDMl&0l+e>uaIB){)$v)`w?v=T^eiQIyvU z?nuIT_|DhH4@XF!w_f)(+3(IbrbofRST(L(mweHNKZUp`at zeO6I)OEOi7^a}UZ(oIgpMIaXkcWZd1f<}npJ&$qF(*J z!u{LU2rdxiu)V(We_3YF=Hpr#3!4Wwj$PTW_I+Q1XDS|9xHzVh3ZAZ&YBV2`6HXP~RDUV0<va|CzA?6|F7Zkt+P$cZDjG6zx0r?6A@eWiNj4-|O8>8EMI? zBzbw4FAr-Zi>I4`kyL8znJi69B|`>)6vJsT&J$RY2-#ILb!p8@1}gP&l=$(00UcNp`j{iXZIFcemU`N1 zKLAbh08%flWM8gsp7Um4UDy(7cR#!p^~uG62lCRPu^tlXZ?MLw0%t^-Y=ckwY9BH9 zSL!KRz%5Ms%28~rT8gv9pYyKn>1n3da+cD%g17c`z-S@$#qkpLa73_fK-=0AdFiF# zZN1CcVEjcfxDJy>>&S-j7kx~ScRU7QPHs7^6BPU)b0}e}-uZ9<&*C?ZlKB`Szmdwy zfdNXXciz7*Sec+^<%>-`cYtJZkY@FDs%iz=p$oYJto3LNhXlAU-tWQ7-@86|wQuER zZs0+b5}BxpjUe%1OaVqhX+sIGWrL}y8uaP-DpRKr5MJU;DV)P4d*}t0V)&(OtcY35 zf%3heXNvCTv|;SCKHJ-}`SU&`4v{`7fAYulxn9LakyM^yI(2tEraR9cUJftan}ie< zOhvM;D9V^6f*27;f-{2(tCQMjUKaaS`PF@_WuS5TB5@c-La>SQOpUTUyU$d&uC!Wc zSsNCfV0|#^Z+qSgegQ4~M1$?d+hfY;bcQ*}J+IN3l z&vQcDq!vm#cPOiWyDWbFPp0sa*73$V-k2*lEqlWhK))dCUZo0)RJPTFv3rAH29zS0 z0p))<14;_l0};$l+(9h|l2)tc(dY3u%zZ&KA@-a{X&-WEA8OWYf7p$Uz{oc{m&-}N~b*%qWizrj{?8{RfL_Qctn%GwsPY(kk1?W;irxk z{FnM<1h>HNZS_|Vz3Bh>D`qYL6CeAOFmeaRMiFs@W=d-vTP$fbv?5N$ zHhF-DAKHR(NCzD2ySeJ&ru&o9D+i&QsU#GGIfWxC*CMUFp3>gq%UG3(WG8qqPNo743j4@aK>;o{5^m zNY_V$d;0}A;F`BYEHR^GPVVkTi1m+BIxWl)O^BD3neZjl9fhqRRHc3vZ+v0PcD~t# zfq+U!{O6B#@gT~4V+HKd&B8V#k!9l;M5+@_ICCbb=5i`C4;{(j2^^h`Jj-%! z=DeWHbP?m0wIG*Y*O~2>?nl2Yhw3N=K+n=rV4WB!xODlr!J2J~i0gaKgvBckb_{*R zrq>Tw0>&VrbYE7%RGMWytH_6!KSaS8GTV1)P^U< zuYJy3D|hR0T4iZ#?Ielbt z5dWe-ZmJIc?4OyE;Ep~tEaPVJYGSiHsNH`l5(u^QySZVTdH;~H1o`l)Wv)_Kf61d_6`$xl(cgZ%(aqXIyDJvPRpQ3HR zcwz6}TKYq9gDIS>qbXslGI*QbJ3UOv`oWChRkIoI(ib|z-mkSrt27F3j@AURT39QyV|LsuDlEuhpu{Dn_dOC;Ev;dYPUpb) zUfWs@3ew|N^3x7CkKPmpy{T3Z(|`nTAll8bER0$zT7bZT)TI2Lj z=HOWaS;f)6e(m+Gpu9o4pq7vJWu0gz9=*-nkcNF&(oFXEbOG&R#%BdDK)1CQ{4xOF z7uVkjVCZA0^J~Ib+-P)#*lm8ui%EI3Uvd4-lzY~bLzY6-Vy+GkcJGCA^erI!NEc*R zlMvLB>uxH9LnP>-FC&(~{HH{u`vKpoeM=<)zFbobri?o*J^JqL!?8f}E_o`ad3{G8Es z9(2!(Q#w6Fu`TZ#8~wX!ejL`H3n8rr=N84Cwvn`Q>S5&`JM9VAf2i;)ZY66UKOHD$ z{+4qp)9GmW>tOlIx9KP$1%!P*X@cp9j9#!OypbU$ls0Q<7@610kyhNI;+)WK9Ynk$b*_8Om3*H+RdLjmPM>H)Z0i34~o{zfuoHQF^4V@fzilu zCp&f#%-8pYdMB7wlJvt=76&G(I z+zb5-Rr*mA#Wrpq35P@RFVRFp;%p0C?b9#09$-U|B6zI0MlGlh$Gk7$hOMac`KPVQhROO@(Vfl3m-Q#5lkxo&{Hs${kBI$gg+<8n&$xvhF=o zd7b)mpM8n0k$GNobwp9UWLtFMYG0cPG~kuofR+;4t|mlr8tlW1%!|K5gqKG}Jqz7> zK4>Qw+3MR&4N6gp9;yB@Bn)@iU>5dnE4>tvZxCbO^plxT{^}e0i1ezT24PuaKa270 zemVxv){DF}5yUiDn4br9r+KK0nOoTM1UJ-@CnVoN`%^oMPnWvu-YR6}zCjZ; zPbHmt^_#p$?pHd_)<~g(ary%M;X0kag0A`K;1jORSx>{>Y00vv7Lz2S7%@Kt^xifw ze5B9ilj(1F1QG#Eoa+wUi+ zXVr5EN8drsN+*94>u?b=fwA=q*~FLdWs+j!qPNChdg6K%wYXl#p@D z=^&SuivOP${N*UZW^t+pb-BP0y#@Q2oDUNUk2XHsH+a2*Ys~EtIBb@&9VN8(3sjL8 zQZ5wmVLW|;;pJ1m6$Y#6V4Gbh>+akNT|UV#zZH=oW0~MFp*W33#h-KgI_U9XR(17f zy?TjxX!CiYTU+X&LP}R>Mt50YSmdY|fEdyb1uvz(IP@gw-aK0N@P8$^Tw2P3pe8L} zRi{#=uXdIyl*#<_%Sf5z?s{+|IL)H*v16=;iC&Htm>n)_uH67B6hM_FP<;CK#jWFC zt#*1RBo3!nSCr7OL2I~y_-1VD-_^k4NH1Kj{r&rWm(tv8IQsDD9TTjC zy)|wHe+-{MQ1Ex@vak8~$9HHj+V5vIm++>S3mLwB05SGn!Z8_LZin#y#z#mVs>OvT z7ji}%*W4+RX7c;J-qmv_MX*;@dx+$oe|q(k7ek8QQ<7;W=b{k(O+)h$AqKEo!1H4= zeJm~L!#R_`xrEfETwHKTNweMY!Zq(eQdAz4$%vIHOe^Dh| zQh2pI6wge1?M;ryauBXNuU(_&qwW4-vDGZ}cR{FMC{x{wIsIKh17@!M5hOrLibuS; zQAEkrm#)o&JaG;Y`lUjTXMSlyu6*duE7vypn^8T|B!Txv=Ftym8ZR3Vv7O^5QMuo@ zfs0_MD%@f;(_2aAqPOwck&IfMb1%Il<)LP%Q7G{d!SkeTUbnMMV3>o4 zxX@rBb8SwSw{6wvF>as6)<%f9^yxnA(Ybid1k*m7VHauc8QtuADIm2JeL-K!{ zK($iYo7oKQFDeTNqmH?%#gmwe%V(=jBm}H{us`8F!NF^452m}nlKo7{zN_TviCEGLmt3G32@Ct9#dI672P-M zD7c$Ky4UKB=q#%UeoLLy&wQFR3Wj|4YAsdy$~{rhD@1h(5BD=}4-W7rx|okcVVw&v zoE+8pTnWp^*ElQI&WMf$vuQ`OJLjaYc|30F($&3UaoA0XVX0_#Vkp?izZ{%Kb-A`^yS7_)cTSMBorP0=jX^ze}2=;0}(nhO*BvIcX)?A`(bA z>}e59%>-nN@*_e9WDBwzcj=oYhc9zR?AY6#j(uQumJM>e)mh=>qMAV%kcR~b;UGvZ z)eA_^){C8HLAq zZT8f4Li<_DJ9^qie?5Mka`0g|#x|Xb$2GHX%)~xjgqggL$G$` zNDuV(&}ecr`TiKD7gQwr>~Xd4f{NU%$4w|Kr(*+Ws$e3R!pM9Vf2QVKrR0_GxB3Fj z+2p*Q$zJ(vgHwLe1?^kGd{m#(U5+@DEqvmV{_fqEZEyl(;k}f*+6;wx=7<1H5 zkhEEm%icft!(tc`-{q@7AfzOWJ1{wqfPgYb%}PnGo8(StuHh0Ha?$g5@D6j6UK9j) zRfV^5mjLrn(==7!ZTKf%QqNx5Pm&25o_n8q%6D6wi~<1)V7 zHg?-aab8Y)7fcGD-?NAwqbrb&waigy@kYqrH>e*W$1to+=eRGrV_Q%hq9dG!X!dd! zvyjbXe)P9YDevWV|A&bI-u_`B*nnOx(#vKf7+TIgcg#-j`ddL_RFkhjX!Qk*PXgb@ zKxldCk4c^~sJkHE^2Iv0D~#_N&z!>Vvo~d7B_kiXm-jK7LSQh zfTbeZ%qumgvT|4wQ7!D=KGiStff2PWyl*)(LXNGCegY>mX&ZsjVI*FOtV^$#OLjkg zA{q^$Z**)Xr6^?#tr%u?SODMggC=7uN$Yb#W#tZLgfn!WZ@c-f>Ux{Kk|)C~&_D@n ziF5PwHyDkvqyYwP)mQ?S zh3~o5V1lSJJfx&Un&4cuQ0M1tuW7MkzndwDLnejhR_(0l4R$W=cw zbu9hr&jv@rrV`4Z6u~eItGWTPMO|a4!!ggq%L5sWh$B2A4k7Nkr%|55;(~T~XhYPy zk=sL#AHL3X2tP5=F{HpC3QQdMKqeFR7c4LkKzwpNIT|fn+4V|Xb*|FgKN?LVb@DSd zK~v^}gsSyO#@(GYNjuDfy7fX0O*E`-<*EkHlVys0N{-qSZEBmQgI zG!5@wEQlM*_$1k1;l3FuZmoCSB!QO?iJ{`M29OgEDJQLeJwKxiuhM+0>bdLRGbm2Q z8}q*XCS_hdck<>^hvarhP9+Ng_8mH2%zO*b^!y{c0d1{;mSDSr7_}d|Iy!3`A#N>6 zNgL13riT3^AF?=~dIa=&4SziYdvLO7hDpR1CDkr?elkr35i|ca(dn$}IHw{)7Y7T` zWXjOHez#p?Ux<7|v1s)~FlWZG|Jg@UZpF?qdMWnXXia|8?###DtR&c73IM_W&SHpx zw?ALJ+Hp!66?=MElfGGvFr`$4uN2}b3Geh>*-^gr`>g>!0Az0DoWb6k{rzx&;PJtZ zupS*&2@d4;9A~6x#|>cpfDUv9Xn%dN;*LF?B`tu2oeQx49M%(eS6pAzsSnRw5{Xjp>6WUa)lXQ+PH40h1|jL83lF4aam_+Jr7 z`$~9dW7vQL-uOHbj-42NsMC<`z>QpIco{r3oM!V=XsWSAvqcN7IY+@36*%@6RXz$1d$}ZmnejZ#-|XV4RYBFUv|?G8*=_ zdewVpXDCW+zR2qD&7b;F`sh$ta&gMNy1A}yed#{{vF#Mqen<7%i^983x?jR^W70QES15ebo((aEyJdyz@mM$UR0j8D# z*H#^QfAB87eFk2%EF1+v>ReM17iXcEoJ{#)dto&!{=R6yWs?m>PA za8;&lv*3*{a!jY0+_Qk#QmrYwy5V5$e*h_j{X?ADH!*B@4w%ALS#=L1Z49`(u3E_- zttM@=%8)9pF-VKaA67usK!g~*zKM(N6on0fN5*?8O^Y#`u_J7u6y;;X!^lFsT3PAB z3CNf)?3ledOEfm?nZ(t{cgYt8T>e(3STD>Y1f$tNzT7;vQ?Z}9g#eT)62qW};8hT# zc63>#^s;U`6~9{#i5j~gPK2s3NtBd2eyE}_7C3pB(_sx>4QysP4+}hsK-I+nV$V?k zP*ilKfBQ`#yE@Ia;ftBS3LL9WWRqOb!zZ!QKxOAlighk|$Sk|e&MJL&>f6pMdB1OY zme0OAEz$hy_6Q{yl#bb&3kdh?84d#`4xEY%1Fmha?3nbq(inCUd^haLSlG!ddu)Jj zWaYw^FYn?U=GsRGB9NDfIbqp$7?&C_^eza^I=QsRG3_BW=HT?&dz*65n3S5kNZ^>O zq1V1Jq6XIdyMq#eXoBZur^pbGU>w+86R<#GkN`g>BqA7nRYs6YDN6Q$eO@G&va|M9 zpM@(>7M_g(|gMRXBjlE48| zKsJQ-2m>K)3E0BF-*7X$PWrbScz}Pg`^U_~2*LUJ?!sZ-m2L?E-#xSoqfwEoz(o zktQXlC&O4A@`BR?VfcJJoD8#>ZuW}`bCqjVU9Spd<|-YsDMfdf_~@Wuij!>7%Dq-1 zjz_ih!bu@EDR0wZ9Q)3+%X+U=${4YHgartLgJqAa1!9N}zXCA!vqt1y@LRYe!KPCY z)0H@-Tm^zQ%Pdy2jrSUjdMM!5k5PT|?EJVq38m9~`Nlr#EB0<-P_lB`RK z3t|dunX11O4|eY|dDjNubgG}HxN~RdsZ_>j4&8; zlL;W$Dj%Y_S8Ji5z0hV39rf-(qZVQMdD-s2mb?zR{gcnm^I}wH=VXkVXROe*8Kg$P6igmphK@w9Z(&Fut*=a>}W6j(`X3x9dQFy3TsT2y+~LLRO8ctXDM(ojh-pcxEduR>_MgswxBwV`wzWa)FqXOBGhunR9G(hSs zFDxouZl1O0PRHq8hFv^VL8Dmk+^nmK(LZT;0o{-@7D?1Xbe&`@OTY-{?OJSrvhZwi$rf08eY;P5FxficU`iJVAIdm~Riu#jfEi&%t zSp<02e#SYA9z$R3m*rb!h4be%(eA2l$+iru!rfUOonD+bhqN{d9M+^8LeCN$6?pQw z0d6?O@^WJhGRD<`nVxiGh`mIFa;dh;9E;W}f(}^2DdBBK2jw;oRaRzys@iT0|r)yJo`h$#$r^aURmje`zPBO03HYXq*nFSlG{d_=;AVtfh)e- z-J$r%qgNoC0C}CIF$_8U;>C-DJdtu;+`7HK)HQ)uY%&gT<~;z>DvrOT0oD=?rG^*| z*@PvJ2ogT`Ij^D?_{-G{X0I9OYJ~48$PVQleE562FeU*1OB@3Kd@6;%heex5w7modaxjla| z_usONHg`+cbeNd5C&IBa&r$n&U^OSdr|F9~vv)#rXD&dk!7#exgWnSY=N*Rf@~?z* zsz36#==wVT7Gb*O6Vg!2a>Q#m8ziZtiPx<$KglEPTb#U^zmj0p|6SVM+;gjUf;5*` z8VBOp6bLZge>@DnM`g|6n80lrb~zm>`yM|;Y)3hdmlX3Sm^Bd)F;5%l21_~tA;zXq3HbdP{673a{7hQqV$=FPlsb0>%CB`#?G z6ohX2LsW$YI6$B?9~g+iFZdPQ&-)L#<;U`DP=>+?DHjOB<%+8NZR28XEv?b`yOZ&D1~R5 z#e&zPoaJ{#buPAo6gzMdK0Qu4N&cSVNMmaLP0$1ZG6z2&0iObzsR`@Sv7xedpB~dY zc%PP!#%sO^a-&j|mW|%Oe#=fCBs=Tua{HK{ghG6O{!ShY1tCIubxK4j(b}?}$zfK| zwFij=5upD_*-8h}XNSHQz;8qFyq{Gi0b~h~5>P$rC22J4g#crm0S8isDn#*&$fQLU zudC@+&D(EtnGdY~*dZ@CAN2CE#CmKCdJxU!;cA<@fr1XE1<+4{2Or|}PLp)7Jh z63Fsun8fzpt1rJ{qtg8Z6bG|yXPDd->c)+kp?71*ft-Zk(-bRlWPr+iHXy}PLj}?s zw2h90Kuvpsex`uma# z>xQ2P2B+yodU^V;)XT1UTr)X5J9u6$9B4s^njkdP5D*v2FD4|`>X6m&AmNQSp_phw z5D{#DTB7KaZSMjZ!v6eG3lRA84TyA3{~?`R0!TZ)+q#Zuy5KIIO2USGcg@s?M-=*ho-R0zdJch(mym1Wq z1Xzmiz(qChoGXwWE^Z96d;8QEh*;N8-dE-Ef9-`F?1jnnBNivZ%i5hhnf*%&QN5|M ztGpp&(oye>Q<(LDy4--B0X3586Ll=VvRRaLgRkhC3<1r*n2tDe)Po2FaSM=1=zuN| z^Lb2Or*Ts9+UvVZ-Bonlkx4&niqQ&u5yD+B8VEq{DV+J2-of36opTXY6ssM*5uvVq z+9pdKL!7z7v;b)!RD{07>jQzARu0Gb*f3dKeSJM%gS`#&Zi9u>h#!6veqsDg451fN zjOf7YeY|W|DC|3}GR+K_QJ>vXUE2tU!=E74iX1{tDQ)d4{q9?Z1&BdAozM=z`p$Sn zw?&^A|L~Pu{=Q8gE?A`0B>pJw4ddM?WOxt-6cGOE-A~?ZvqtoDomWM*CttK{oS}Dl z{IGhi5(M98DLW-k{Yh)pcmN)CADq%YEOmi{z)T+SHDuT#1wKA-VM zpj`5|&xLk1zWHss$!Z9}<6=u^v~dqf03NqFxw|8E(Y0@TBW-%_adE+*wphok-33ez z3jka~Gy z)eHL3Xgs{7_uf2uE#0$iUAUnA7D6wloxk`*7}Q~N1ZPj6R4;ecJM;b*SQ0ecf#zshW_|L38|)kuMEiSm)g8ba|X8m`paNt zT^LA=^+XVt9|1lv;Qv!8Fp7fz`sD&*CpC0`Zr{M}*>u&RyfnS1J{1FQK1VmxfC_d3 z`mSs*W&iGpEz^hU@gQ4c)62ci_UU6`b#%K{5mcD@GPf0Qq~;Lo|1K~X63nJqyi3E^ zF@opI-dyAU1{m+cNoFZ^ZynV2T|S3pukAZ-Bn4j+w`NcNW>s4@pi*<{54Q;Z&Q8E# zPPpqJ-FfFAS4UsZwMDmWa$n|AHORR(2vFx|t7<(3iHWBrY)ffslGYA|&mH{Z&l@(M z2bE}Goj?_(H-?n(k7&YvyEBlW-XLGxekOPjiZcGWKh8JkAkLUI8u`<>HGpCkxTr6@ zP{p8~qfepCA_WTZt?)y5sZ0;9|LtRdMOYx&BIo^%F$kl6Wpm9|?+hgJzx1v(MR&>0 zmztOlJjqnWr633OkB@U+SuzxsX^NvAA^r~Q;Xp|oPR9f;Z-pQAssq7BK8AWvUJ43e zggookuVHcNfIA(xogz`yV*!4pQQYPleBdGs$ML~X+8!d*Yt}AwuNI|g_eHxqyq8=c zcQX4QX@Uh10Z{s)YD5C*4Kj-Iz3WPE)_ofd{~4Y2XP+)l@=U}?5G&j$D_!wxIc^M! zy8R(FX`JU~IA$M!4f>1mj;#n`jKU$q9YMY{S0fgyV07HJv(1{)eeRO9b_NIgN_0YT zmmhS(4MDN&*WgYJS5NYZy2nH7Wz|~GeLQd7L3WRH2rU(DE<9lQjgZH1w4tP1J=2*f z3>D*Lf!T10dKJWogPa!r@NKTAYIv}?bv<5^vdOB~p%7lFY2+%!f`uQ^cZ|La5qT~Hp9&v_W@DC9s+yoUr!*fnGvs+^;5`}R;r;r>^url`K#>E%DGgw$xQ zX??R7Do5Fd{2bm0hXEr9E~!T2oui{bn$^rt;@o3Xh5o7!kWhV*?U=KlT zBO<=fK=)U93C;w#-B|ImDF}lwhQtk;l6In%ldpD?r(cP`jX{4Epu3;qWmCTb`LHAU z2z$GbQ-Z#E1y5#}96tVoq{on!pdJ*dtnykHvsNp_b*nbqEO=Szh(oq5LsGxt)#sau zkDt_G6aH167cTt4HE>G<$`VIndIZ3wMab+Rj3T`}o5ex;?L^9)^5D=V1nZ<8>f?U& zU$*8rZV=?O6)`$tbu<(3;;aW+EGGx#;d@9p5IhAw78?mpJ@oD%69Ie}amL|4J`B{} z11#&fS{Al;a+|23aLh`(H1SI{XPAf6=QrnWN7ZFEc*B*B_cN6jItG570eROET&zyiQD0hOe7FPI=PkXZ|%#Z5m~bpWtw3!parMB<$WyKtCA%Oy>;rnt3c3a)8s zt$brS_G0}Z%mzdOB8>K-F=RR_ay7M$o!m7Gh%1y53lP*-iPXH3jA z;x;V@g3NbYI-+w?tT^1)^>6;dWB_3_aBHD4j>mBFS#3uiFpT;0ThBPEJ;0-?pBan| zN4z!>HlqgpEKQbSpc)X?(G^$@@|sW>{7EJ6U)=M8-vRnk#Ek(`5Q1c)(ly8Xi>{eR zx~sX*7rA9c30c7$30Ob}$dV2?bdLi=Qg1{{@fW7q#Sr>JtA3Zxxy@`1NlDPKeh7VF zV{jrUT}>AJ1+(lU#e3oAiHl=}(&TzAb8hYn4Fb!h?0yYtRax})P>^IcV+T%;uZ5st zPk+VO1QI~k@IzizPX!Y0XgiGSc#2 z5C=`4ke<3|m>f+_IZL&eu4kF27~s{}L49TC7SS5$80d_f`p3xvZ_Wg9M*7?A1ABP8kRh)D~7+BBP7s@?7 zU1-_Zp(fD+8l_~TE&uc2@g3l+_n4|+(&1nOHDn-pI%ZyNm85qq&g5B|wwQ!5rnblA z6#6d&9>*2D4v1?afuD7>+e@Hx*Bw>@3ZB~=dHN(_+UPYUUD&W~PXx(dUL8>UY^bKt zu>xZ_BzUGOd?Cwf5LB86d)z@5k$3(Tu^&!)g@*ZmCE*k`AnXUy7db}<6~-yZNRxW6 zgaXAhcTVBpbwe6p9B(Y-2@|2JPPhnAqA*x_=sCel0LV0ku{U19YHQ5}P)SQu%V{sP zc4vsqaUYu>vTuLet2)%7E!Dd2Cih1g!2hxo!Fe4e`LmAl-SfCOCBXl*>P+ib1+CZ( zjHsp(O~=IRZhy;B4b&u|y}KxfsbYd6xj8aZ-R2NSnh~O8{he^oaPP(aqj9oYG7?a{q?_sRHF>un!ly1O%167 z(De0ASL2m^lPQDoz#n_B)h`QaurdR*0GTh~`m+WiG^9Gzu2e!Y!W8?j8waFf5~G^$(7x@`L3 zE;I0;+HXeU_K$RwPDJ0ZR^GdN=L0Aj6U-GvKk=BJFvS$A|0pjBIKLfC0NZDFfV;oD z?6_5XfuTTe^UaU3Y{-SkTD4^3>7G1agz?#ho1Rcf1BL zvkUZ%bl!3|#Mo1#)G}(%9l4l2TTj&1mqv8fQkZ?!h*%t-0zj~!kuNY@O3oCc*c`SRPQklxAbso7FqN>-5RhEXO83s`JNt-@ z2r2tYx@*_^V-P7{>}8#-s6+paQpb==R5i)W>7Xxe$z$2TF2(hrF9G8=kPAdy>y$n>*P{u;9&qm&QPC5e)TMr>Ht0s zahsJQCR;EoS__)2FIF6Fy~;Fn-qo{Aw@#u)qcQm{cnnB8&TZonwAp~R zxT<8mZ_GceC})d6AHRt|VgoJoZ+H)_GlrwBQF+>BCSe^lb{|H4XMqgl;OYYjT8h0q zH+owOmciu&@#_-7d{4O$C*2xbvxc<@+fu_mj@VGkUfqn%wbY|=zR<#OT(43uyHKl- zwYN&6NFyTIKP1k8F|&KhWeD-pc)6u@;Cbz)>l)H%4zwUWtFtWDJqh)$YUY-8+$Q)F zNJEA%UIXUpi#{!$u-11nJ{;ra+wHEuH>Eug&zN*9n%2*z% zq#wy~wKvJ+#S(A^yi~^for5pYvUdvJQ>{I=)fYlMD)cfswzViVrlD}2 zJo^c^`OA{%FxS~BP~idH+2R`KG1rTK`&9P|;T=?VdxfEjd!_#K$LudBEHilY9$yhd z)-b1AES}=rneE+D3R>*h+tHHG^hiX@o@IBO=X=dF9O}N#Yj-*(e9;Ac9BE0@QfL<2 zzZoP^IA!c?JQK;OUDs8ocJsW`P<+E&_jGSjJP5$6C@cqJgB;k!U%8GFLK~V2FiISf z`rG&b9~(F8vro#-qA-tC+Q{)Fe+@1D&JcT2mPi^Z?Me{2!^RABBb601c{@Q%NsqQZ zD@efKBmq4!6-~fr2#sQXjZPqrAqUM1R<+)71YPGB%ciTxniuvJ*F|(RDtT@f>+*}O zN^JEsaL?V<4|~7Qn1KS?A|Gd+k|s7q!#5{s9Zmhpf?LoBG%`bB)s(i46S2eT zE1+yOGb`FnL;Cb@&)^tNH%ow|OrQS&ub+xX2k;0S>N0xXD|d*6S%fjCCn|BJr`6{> z=zNWPA(QB1vtqCJneXzhqPkmMY$`$bNK~Lq-^UTp3*JH{G3ycpKA;uEW2GZ@be@zZ zK`5R)B`Z_cT5>8;(0%cC{R4m~FG$!mUWRNrW#mZcs~nHoBPE=}c!XyE3%|V(2Ya`N zq!Xs5xRng{Xv?zd2WmwYqUuDmd{SudxGbqV7#7M>6A#dUzDrhBO0FF7HnX{px2&9~ zNCUoI$e=cjT@o`6oHUFG^H9+V9i&*x3$mx7lX?4CKTt-u{}XDnt<(o|#bG5KR53Wp zJ9NlRk(Y_;+E#cdJ6p8XZ8yVyhO>PolY0esu=m@2IF)f@;al#^qvVo#&%L&T+ST)4 z_($=koqSZNK`(p<%Ib@r;-z7bOpL(Xr1<)uUg{nxWYCtE%7&;pcNwy{?Wp(CHN>{s}(mb|A zB>QK!T!fIINC&#AC6_k`l&Y{@Y8V5A`CWe?QkH(K_(g}8hdyI286TY!3u($( zSsHj^)d$n=a_5WT=z|K4-p;*8_VJT^ zdlmWXAcA}#_3oo~_&?nOpwmmQ>^rk{CNI9MuF9va_65cINYTg#Yt`u2D)~a`u4(k6 zy~|H~UTvE$Sv|S-bYR=MBE)^(-R|^cm8?m-L7T%+o0`PMsf1OL&I0Mvw0q?J`g+qN z6`)wA?gkRvEve0`=E+foV!w!~H8jaP>*{xwN^jdt)w?iWLsi?J7SmCX+6%Z$8mnKllsTGUlzf^kK$r{;w2t}YPkGA*I|ht*0&az*O*x6QjX zvk`fWU1qzkMV*1q5^JQJd^|Ubo%TLKU5wPL!c`1xVg;Z+Xib!veb7%M)CV0G@*@w3+i!i}v+LIeAfUO>k}^zhL830M z7iqAA%qiKB)r47}a@VH58f{#WaBqA~{qmm9p7(qE&&-`=Abl4D$D(>C9ac;oLUTYUS&58+V`&dbB* zoR@Ep(WIagAKAF=8amu<7=At=4=hxQuj>-+RfN_6U!!7a?}K`r{^Sp{;KRLm+vsW& zF5Yp!tle)=U6nhIqp8dH-tEGHv6M(@b%e3zy5kqVcbRZlxhL{hA~ zN_rf0j8a61#rftbH|t$+xnu9dOYp{@N~Pc~k*S+rlsh}yCC}7yP|Hn;JB zb3x`6A2q7LO_8m&Aff#T`#O~y(QsG`* z_5bL4%dj}RtZ6h@u;A_h6DlxcZc9^!6CRi4UN0IySux?cQfScf?xrIDGW zyYuYNPHyAJ-Z$hGc(j}Z^w@wKFt_N+tgK?tX3**JCNBqZa;|(^dQ!AbP#O7Nf;b(b zVRG)KUZ@!cJfjp}G<4`0x$h#Rj;_OHo7bK(RL-ubHU{qbG{NN?Bf=L40`b^W8|>%f z0-U1)g1c^eseMQ&sYK1;A4V$l`*E^wt6RE083QaZqvC=RwJAMVc3+ENUEY2@(pnrs z9V9Q=bv3%6(AO`A_VN_Ua@J(}qWc~u8Bl@Ogiw|XIGk}{-Pv;lJX^`RrY|g4UsSM2 zZ2!C0`O06Mi?xoEms&z8dX7|=Hjo+Z{vRO2V=sJOkq80Gs|$1REua+<@02Z^t-W;3 zG}MYtFdg?-vYB?*Ilew|Q{F$}d%s@{*rY;JQ`63N@_jOBf4q0z^If=vu+(9kqMj-U z9t0DKZK`@qiPx~&%FZS*^r>ELO*JUm>uERfOeos}4nyRf`-WV*PBornI-r`dxhkkH zvEQ=F+3QMk{N@ z&$ID8Y81ST5ZZrSA=v)d6jQf*d#dAoR-C{LXPnf^wDRb?_%J47oyX=lK4vpNxv@b^ za?&c$@-yNG->9wLSt4@!qsxoTnf&)YPLQSd#Pp7u=eqCTNNk{DV}&lJTYvkcbQ2-+ z2j}ON#C%>MafMkzDd0Lp|1pb|``PYLA?N3HM0@ksLHkvV3-cTtUZ&&E)W5=e|19+pbP+*|lVov9-O<8I_3|B?Lf-189C%@3~MU7m|{avdW0bdI*&DrXdw zK^~(|J{^pzm<&lT=2v5GI@mk6TeZ zVlk}A*{xB#gtsf3pBKn)j@x&S>&;ofh~1lpLL))Dws27c5K7a-5~%R^KE9LR7zRiA z3&qrqv&?0Mnf9C*QwX32(-}9@w5i#@dZgYsxX<8s*Vn5qdM)lw)O9+iV6MO|Me#^{ z#p)ot4=?RWzX|V~N>(_I5{}o3UAb(@B8MQ&i#YmWP&{dlsZxIK5^|l5)>}s?7_S~D zHJvw5@p%eoNIt!6vrbhkq*9`Dq*Z`RB9C!OT>)27b}f=D?-|~|d+Yz;F;TTzE6FW4 zg^a&i@0QbL?`HFGU)dl@FQrQaAak2H1cyKPbMGE57|hK)IjhY6u9R6(bXVtb=-$Tw zn_ESANG4iZXCRiO_8j!NFX+Z^r*fh{`^iDqFLDUQ$;qJ} zT}2&^pAX_OsM0-VKaB4C9%;ru(VLk^eB&u6NC#^-k}P4cX|_$$<=l4is95=uUp#>8 zN*(KJ8?%~Ak#i9n0O0!a*rVe@G4pcqbL`3WDo3DH`%EA+tu{)1^P%OjA8Gt_#o77@ zwe)shXlgmDQeuBw-817}V#PSzI)YPPT6HZE5vaoI9(={llR?4)ltK1kB)sCKzh|D- zk}8fPb{-sC?!Ahau(d?@kpj{!RM} z$d%%}#KaaeCKq#qwa|E=L$;q~uL7Ptb*9%BZe!=qaLvLp1Uz)F@1}H+QwyIu-q;Jy zd0Z?5kRNTSyR~t$=gHiXVV4E;9AdVYQ|9RT<~La6WZ9zb#R=*Q!U=w!tiDw7|*i@}1hqM;p^}%cxPn&S_?9ZdB(idMr1hOH)!r ze&1fWqir+o8Rme#q`TiV`*;o#B~hHY8;5aTqRqaTst+vEvaSY61*S9r)Ij@kx{G?5 zXU~B_haJkI&-WR-xv`}Q}k_D2?^rdfLEG zKFrWzbKSg=c^=z1G6nreuwvJkKvX{&+xA!6ZIbnC6hqiqlF4j{B%Z0)vP&^!H^+g9WS%qw@(qd^|QI$L__?0h} z!vwg?2MgG%&&G^4=o64MWh|(@`x8>H-10%J=V_dCk$yVoVR8EDyJ&)e%T4)C{X~KzY5USKLpz#v-RCqj=ATZ*CV7t!fNfvtp{LCN zxYVr$%#$y#wa-6*{_gk9ZsUN;^O1UWEoN8xeE??u(&m}AoPr)%3CT0-k8fQyNCvQ} zQ!A`|Kd7BHZx*b=?ZQLL5bI3RT~~IGv>ay)#9affi}dH!vv0Fku72BWu6+Xslzo1) zBTOm-qdAF!7Sa4LSXT)uM6s-11@Srre5&?)c~fvvXW;$*ZlrDU>5bS;4In95+F_qZ zd0dAy1ab)fqCD{OI)aSHw7x3UY-*@|vT^FqeoRW`JN^|$g;qbHc{zM}t#c+N!e0 z$7U|~7losa>Wf?s#2Plk)a3kV$!+7idQ@@587i<2e^HYg363wc&9 z+acrkLQWC=0D1TIlF>a_wW9IdoWa`EwEFhe;)_P6254!wF@)oMM7mmc9(3PsIISge3 zGieb|E1n4xR4d#Up4rtOU3k(|$~@1s6nYzy1$G_gA+LOMsHT`MX^Sm~{%BF*b}fui z-t)b|QSZ{LIX>)bu<@$<4>CD&yIZ$a(77Akqw@ zg5#KQ0T!G1$(<%?VXWiRG;~2Pn`US)6i6C}mjb9W9PkPsE~j72e?cb~g{2m|NSM=> z^`>UK90$LYd~W(l2S&p9F^jh^wzyYVS3CS@AhMYfK2D|tP%?EB;cA@>O{Zek5xmPO zS$lIuDg(b-l3sKmK|FW91?y8u3u~_lx^nH}f%@eyO)h-ceV_14q?VtLUf6rHA zZ!apb#a|EGMIs%#%n4T2Y;|7~0BUfYGKzEM~= z{SWH0X7w9^Fnbs+^Bj{qCY1r-6+6Ds!l$pQr?g3qwzu9E`y0*OVD2!QmvL;i8HROL z9#IO{+I<6;8Iv)1P>^YRo>30g)q~W|yj=A>Pk?l}kif#I$z@2GA@SS|sG$}@ia~m5 z@^)9ahSkM-_Gs86Ci}6{u>+YTB&B7b?MET$KF~0KXYO%kZs2awu#mWY zF0tFAu1VS!`q1b_u1}EPzomXatw-_3L|CjmbUa^)a9#E;?a*O&%v3?bOK11QD~WxVjs7XfyMkK+Fx;eY%!O9Sk)ptCjGPCyM(ZolV| z1(x`(Py*xP6Zj(!^{&Fs-D;?;;|217Yg37aYh#7k9cs!v=ec>KJviz<7bsn@MsBN& zmaUK#1I*Z%TE1W%d(2rdt9w?f^6bjse78po99Hi+9?Pgm1}J^LxEXot!if%hy){u3 zxdZk)Xxk&pG_WTjxi?h$@0IB$FT{v@wS=`Oo^ZO-Wwct?Pad1Q6!uZ9=NU|VN;niU z!_JD@BC0P7fifhqc_@Piz>!$1MD*J+`z#0GvuVLhrOpM|NI>CP7TlQSBCmdwVgBWs zbt-L@+F%2SGMF5mx-!t4DEVY3gLI+P_1td-P-qT~2~8v~GWh9@#WI6F9W9hyd5<(^ zj8~3wXzpHG52|MZ58xw?k{%#(A{+UJCzAqc$?D71Nm!#@gx=Bt=*b4as?sAU0F3`<)>qzH|xjB4qFX|T0vo1;%6oga%JUX3~QI0O@O$WE%PmHZzcXr zqg_s8vQ({`axbs{k0a1VuQi&y)T#qx{8aHW!gl{p#8{qHXXB}FAEfTp=LabCZ}X;Z zo6DKcV-~pcWxWwc2TS?>6J?05!dEwh1;J@%J0^KW-4Sc(%|}*K+Qo0&PQ5Rw}&cb$IWAd zr4IF*bGfd#eeJBR$&BEYW(y*JboYF%vjU1n z-+3>B@IVH{o&T~RI#zy)W9&sDkRM#<9WH`J(cEXt(3is%e~6D#UP>4D!$FS=u&z_hYcA+(genFO z4B{w@ccIvC(@$m-ixxO03k<)PR~V@$GOi}*PL5epBwmRvXK5Kgn{o7{NS_R1U7>9X zE=`xZhhDXr=#{{mx1e6R7cb*5PF=UM8RyRL@n}Cz?KfpRP{1O)Vb;L>@`y+%)L9yv zCl^uHdIToS zY8K`k{@7|SYOE0o*Y&($}6DNI2Z`l&Z06Rw7K%Q zmAD7I-O5LZUkOOYZY(}Gq)ZKO4y(ID@azR(Zguw>nvYqxL~iG-96Yzyf3OubPtRQN z*}W!}(mN)nY1lPUJx#@ON2qZHZF8i0XrKZxX|u}iT0o`0Y}Yx!Aj2VK|BP_|$XuH? zqW>VtMO)mM2fb+lbo!el-{*#X!@2uIkoJ(h#eo+-M;y9Sm9~R-6+olGT~TN>(fo{afpQ( zqx;YJWoP`O@c@av<`(1{)`_*x?v_v<@Uev0Qq(fY!bT7nwVz3BP)g>X|~2~h6zeS_Igo8`HImR~*V zB-DHb;|=ZuE!C=5=*yWdL`BnZW)gN03A8`s3!Cfj6Ga;?88w;+*rNu3eieeLZduyO z%+y0>c8}JI=5tsifkiI{XYWkjL4L=%Jv7pbL%F_sgle+QO@A*DmK;C5=%8{EFTn4q9>Lt= zI;g(2E)vbt$WjtQhWmFsE;tWe%&=a|AVRXb^|UE4e&4pD`q|dxMvScn@iUH~j&H8a z+98R54dy7xO=*Lz zSWAfM9;se#pk79zc-H3`=|fSuTCn3r@M+QGzB!jdipZwk`=1Mu6@obhY0S{f7Y4>3A77+1zGlrcyT-<*+&B&Dgu4*C ze~Wo`s?YD3P*IHXL9j01PYYjvh8Li1ngHGTo9C$32nEt3bn_%WbGpr$E+Y)+j1-}S z0T>_crsMgz4&`vTaMECJxxyKE0nYgW&0wEf~zhtO{J+#*UNN% z`;R>0vi&(%4t@*StOdKexB0%)fV;H{?y=AfMuIowq>9ka>E{ID< z2pa_7qa;U8oA5plqdVI+-5Y#Gnb9#_en0WHVt}s%Vw87vij1=g`lD-+M60 zD3TX+D{2iPFKCX;GoQIdfq=!F;6x5P9?yYbtsIlNF!Ud#;fcPCfUS54JHjkW5$HRL zO}1%|F&5bz5f*Kg=>59Mm}*c8AkGQ~tti4?q?G;ARBp5wA*-ae?6MDA|B2J{$yl)N zHHGQZqiN}VX3zTz0&tf|n?ZVv58g}>DJoaBf_qcz^cB- zGY7>gWz86>EmN!*v|i3Le8tEN$^lYDx3ioY(t5~KIO-!i#LhnR70&X)7kdvS{Zo`G zJj(73%a{QjCCDzx286kBrA5ahS?%|v>K?$o?D95w7GRsWWH^?{n@+ofep_{YLp0*zjxB%I$tg1x-{J;)q+bp0a8Gu&)4uR zbxTDcHS(+7FePvWaUbHY4`AX@W%}%%PEn>dHNF+wOnr^3-@aOLUhkYK0yAO#bQrae z-L=3rGhE?m-;S~s%iNPgL1g&X>Gdk3dbQ@$yJ%$3Hwkt?Q3qKKAZ~k&hv;J8z#)(| znxghnK>)dk+ocQo7tgmE&5H!?fNH%*8+cWP-b$MYCtRw`e^G=I0(+Im$TCZmDbFy zMNizPpauguP+TPUsf(cm<*be4iy)Gv%bS5&5`6JqMTlD+Z{0S2@ z=>o^-czpVz{p+;~)h<2UcQXeeM#HX_Q+b=zwxtnV;Sm2ZfOoIKO9JVx!zJ9aHi~AR zGaNs3a&VZiZcgjDKx){2f~9cK1UI!nP=Mf7wZ~Yr37-z9iX|f-{q~}~J?eZ)Vbu1XtLr=jK%oEF z=RE8Ml)^Mkt9L2dbRoi|S}?DEn>}LCea-V&HT}kU|6JvqD!zJgb55MibyQw|CmG+( zco^H<^8C{|DB&dTSA4gI$ksfbUMv%#Jj%&$#dm+v3j>G5`D+>pvB0CYmdnNlPLSi( zlfgjYqh?4TaA8DzzYA(FH^d1z07%hWw~{k|J^?@U+heh!2Y*php{?pBp<(SICfgD- zfnRDcd2zVg8Gp!`!f{Q-cnFVy9ez?4zF@`u56RQZtI=+Sbz~0f!I2^1-jjr=wz+aZ zE!ccCimTx(_AkJ7e#_1T==uP&e{J@DF(GwYA&kvZo2Ikv%YC*DuF9AmGM+A#5o~Dm z+S4!TsJxH+y!Jy^=bH^91F#GHL3H5@d(k^g9)h%p0~`1wr2jd5*IN5Z-3ya-zdqLkcN0|rQ1)RtEs ztjlc2#TK7U+;bG4@ZsK4qQdp3%{Pr7s`9*G)HVMT`0%H;5(ObyT(V2WlzmB>D(ag- zzkbMo?gVPqgtMZvmb4e08>C}pjZED*KNK&Y`XN?pw(C1Xk$OYaCkt!4A}1N%NB+OM5T6Eh`OLUvk*ZTs%s8|1)oJg zh3v>PybpX|Dh{6PaAfLniTsqWIoj{Bg~T*OKSDC-q{_@M5;tXKiE`b=^+ zIlECJ>;LzjuXoC;r2Hb{H_Yo^TYv*MY007mq#rdW-oLRP+*R}!)~T~EYA-Q;oU>j; z!#|&x@t{B{ai%>!DUf2R&MIy=Ntd91-W2W3Gd6Vyle&@ ztb^-Li=uhoD^k7=FQO5S6)8{+y0 zUs%zbh?@H(fpgsYr%eK*U-hzA$2xI7%T3HsInJ$H0SvKLgZDN`S?K@YwE_QFuDwXF z+UPUFiCTo0b< zs;0Qn((|FLQsc;D%H5=-Zn%tF#N*S>fwf#yo^^VP!%B`gfxm2B@S=Yql5nP<2SX&} z6|J3f=aF3q|KVVn%gH^I$5oL_fu0tW?)T5I1vR%#>#Ak%^~L`P9P&N2^$!5E$ zzFtGkUyv9GN@B!0nFmsB1c%;hsT~pnViwy2H6T7^?x&Vb;5%@QWq|h2nWk~Y)A%|B zWXjl>W#Il<$CO96=Nn_~^v~8SJVzvXxJDAH@6@{%?^qp(^=k(0B6ONixk271rWP?mfA^f-*>uXErk3vjW3mc9$+O+8A&^`PT)wze zUm>XGEr^T=y43)iTXfD@zlUrKy%Bw}e0+BOjs&z;;fKx-)a3EL*DmR}U zKC``KxjtP^-QHJyd%05hvTUrBhbf(YM^;8|Wm&zJF~*d=*|^bd|7W>zX|q_I(jj9s z(N%?KIeYWSH`UQptHy4%?v-=0#Kt<#SW2(IA`zE)y|gt^p}E_%xV(PAXVeIhT+0kh*E^*vDcY5JTN33!p>IlqYp3BsV(W zcfpGu|0+K!-$H$4xHzhKYLFP+Fk>!&*ZI}MIIiq;L1F3Y~=NpU) z(_%AmY-5IU`;l)brZa79TKk*2F3WyYAB<$K$_QwvK7pFdQ7xqjG#Nr$tDA&vUp+-Lf!VGvw zxj(;Xe`1*GO)lTM)Nqt8eOvPms|u6VO6uzT6szqQullAMh;Yf9mSWo49b zsYGriyqTV>9?-YJ@~zjpVX(~?cv4>ckbRCQQv4vAHT6rL*?RvrCYEvNw^d?OUob8W zjZIv*VWX0gJDf*-`N^QYGTq%X0UI>?)#t{d>JO6NXDhMgILu~K5}nB|uv_W4l2G-Z zHh;SXTd2Q1>VG}>2hvu`(l}1Ds5*}@abNX%e!>2_SlAx2yEA96Sm=*Vnd*NWNpanT zTN@s2S5=GS;;ORCfa#1*p1TRv>chETt`5Oo_>-inFum`VKkgTh1~a!Olj6z@jQAI5 z$tU^e+8`Vpk@J$JaEbm7iE=5_L-fc-7~g;m+f}?0asBB6|BuhBpNbZgAy~<6rF3Nc zhd%T=s#pK0BeGX;EOvo0M?lc2)PXyPpR^c3`j6@Q&+jlE@j7P>6EJJB|0Fn6R@PMF zVLktW)KA2dngIz8o|o#Y;FJojQ(Z1?zEep3>4M!^N?^%?j5fCkwEBQWj3?2GOTzet zic&`>{S0D)84|4)-8vcT&~Uus*WtX$&5-YDi;h4?mWBz0G#{_DAGW*rF1s%am$H@R zVN;wYs)hS3d!Ur)0ck2Jc>>&$T37Ne+tv-S!W&*HIV5 zEr-$Ud2!UpoSs&#Mh$S2w8iFYMY_47P_FPm_T5ByGhajCLZ16NCR(m4A7O=X;l>z} zd92_{a+ZN0!>bEfL4eK2K{(0yaxb*m|j}wCE`UcYXhkQ9hnV#RZICvAaZo@PzD379byKKB|@44 zZLYZAalH9`@M(%i=Bz;6Xh~d4VX9hJ$N$**fPFBi$oj3}^gExRjRVJO4;@`-LAoEnfw^t(H)V`$I76-=4fFn%A88@I1kasy0`FOhxeR^8?^*$6NCFM73aROOD;9l7Bwz+1787~IzBDM#oTqBjqX}c&$ zYdi7GxoQC}u!{Ma2KVveD=2QEQCI0Eq?6&xA5QMZV2~V@9d>BwX9dy zj=Jf$KjQuea0pAayIBN*p6R?nh!Vn%1jQ1yy)FiGez#5+bou0|IebJ~!fF^E+Of_?RTIN9=EAG8w)V zZ2}xl*!i%!T?=qn3w*Zf(@R*ilGs(1IE9ud2emnGmvb6~1ta3rYHA#_ZkzN(7Kw8J zQM@Q>khw)9mB`{X3XFXPt{(xKlm|oidK&B(p4g!y!+1yE&VeGOhs-X~-rk?zo3o1u zNWTtU1ULG9^@ch9K@U6Pb7hgn#LAt`%a2H-Q08Siww)LpH&ma6W4m}fLbi4rTRnng zD7({~o)PL21zZ zY_beUKNf5*N|f6>A{F`lVCIC|suV>YSvAaL>$^#AKaV9&76W6tC$HeqRAC-K`rUrO zO{+MG_|s3lLP&NDau*?6NpQUgK3Dz=<0<_j?gc&qyy-JLPnN`RCF&z&q3w`pe@sZ^ z<0&+)(+HV2UdS|#v;V~=n$kPo3$>n(EsH6$=o1ncUp>Y{^~G%W_Mq>OQf8s`OpOwq zj;EtBk}NpYW?g$IF-qPNs>BA5+hy0*@E|jF3^dEI!25c z*5}waSthyhLWqMM;AvDIN88rEltZ9{3cC7sh43V2%lOHL)8}|CGTHcce1TX}9tK6gw3hD+66$9< zlzIli#p@fg=TQw4Pw;hmbhZ;iMh}8Yk3*FUgJH+*(~8JqsRF|_g(@wLP|z&vD$~$r z2=JbaWQM?T$3uFDUA0CAe1KmTM%-^Q=AEdbKV555 zPtvp+=04MekLPR#L)XY`K?DFYp!9!&Jqzg3N(q*nT1}80DH4>ma)ap`mHGM*lPb2F z0R=8R-1TQ5+v{dGAmrYOO~%dkc#l=h&1)0uy^4lY@{8td9X(J-0I~#G79QJIPaY26 zPSNR&p-iWRs|1dKHSfq{4cV@M=-hYj&fN(Qdl!qf&b^?T*K;vk%0M0gAlPK8AStN@`OSUhy-3wwbSFTHbmP5=?+e(#eNQ^wAe> zXS+-lt?Hl_;Dq?>N#|*RsghA?esLDy=}$>X+H|@PcJzn%22#Xsr_SgpmbJ#cU3565 z9UqK0$v9`nYwvkTL~Z{Bhc?}OB>7#f!#VFZ3gL?0F}2rXA;py;?6~>0mOq;6Zi1vd4FE27Ia5n6X&Y*pIRoM{o+;YAmd7C0of+Aoad zpu0(ukgMd)UihT(Vz$~M82Eh~A?nx0V?NS)5A%iipGI89&Pc!dwn?JJ{8?|Fx7_;p(^`k;$yHo(teNJ+PUk7w(7oktM|#X6QLPh5(VC7MM{x)W4a@i4VY$(CVafEM)&y>-`g9h zKE~KS=tL96L1y!omPvJA%N!@mlr_@r1Wt6Q#9VmQMSU0L`RE1GH$ed1pCc;!u#l6$ zjH@q#Fn$Wqe&$*g-?Z?4_ul{5zeoL23A5MudSuE!h~U4rKtXm9(g`VTBPDv# zNFP;-z`A#bHE*%XzpKGb((i3M0Cqv-XxH3*A_g(b&GdY9v8@c;w7Y}VDT|94@vQAh zW}5fdpT4yy{vK4yf4F0NfR`NmZZ0Z23bPmVs`jQuN$Ce4{wSoVj_xw4ACKqjcn zzjbpm3~gk@q{w3aRvZ{aCcR#>hq9`5Lh;>+{Fi?Ks*O3Cn1W^2B2@+_Z33 z0#h2-^IE2%B7P-;i{##jlAGKtz3aY`T0+(#&BKJDUL|naqH#wvlACF685TUTOr$8G z=6v{tyj7Bs5iJ+p&nm(=s$grWd(#%@vA59yr6~OL+}1Cl8KYFfGx);45em>UWgJil z|HpzeHic$M?S!Fx*Wr1^-1ZwY!W-&FgK}81iJ+<6zt7&2Z4IA)SXqeu9+b=&X{IS8 zmilM=H|@0WvYU^n8ghtfn_s5|%iALq{;IpjY~K1eQN}y^xwx1*Z8ugod8G-3E{1GD zGWB-`;roG0m)w+u>Z=tq{JnI;-DO~88S#B))fVP(&0!C6ueIQeF(#(PN} zPPfM}vgECk69UW}c`l}6XpV;>T2nQv3w+tm{orN($C>jM1^(ume)p0=W& z^CE9~eiVrWNP?SHD`iMo7NUg24m;P4b;Y!}ZOqaJ!q9N5OnK?~U(&Y`pw7IPZ>RNy zbm(1jv7~Mtq2?HVhzdI#*iQri3hQN!%?h2quaI~Pe>C=Qy4Vk?(ZMudiE)EB{O;ch zDDTt?=9JYAg%DU_F@^m%Jps|{hK?hoR`l==;*NAPE;$K0!$m=fyfpJqd8qK6#J-YJ zVlW!NnzP(e^i6L36=WC&H|_KvE1DXz^8`tK=12Unw=2K&au0BG`~u)Ts%>1~UI`kX ztgd_w2G;o`G}{*(Xgl^ARK=KIn@0GbZUgbgXW`IdY|QSr-`Z)>=R-`Po$!u+5pWw{xd4HtDc;zopA+PMuB z1w}V`=*CD?1ZSwJ9j9q+n*0xmyS6f2{e*a%L-e$kTiFd1wyX8ys7|7#_Oc$c|i!x3lGWLX1u4a2kI)D>`e2c>t^Yl{)qjHVfmwcPCIpsEL=~*%179d zlQ}+L$D9ZR>3`L})HZtp$6ObsaKa%8=l`gbdXeEce|0`FDArXn{3_WrJ&cdisG|?d zKZm1*ZYx`i?pBVe%RG5cX70DItxBn@8LnH$$94n27HAzME;C)I%X`tO^(xY-LcN$1 z7CZU%8_y8wnU};GhK#m8!`8AbtJOpWEi|h48P^Lrg_ms|aai3YH2d%M6aoBCYAG>q z?{eq5YyuV$yx9uAGQUI8LX2edE37dpSomI&{OBXH{0;|BTn6n*fgz^`_8QS+4N{Xj zcJlD+u@HcxJlg>7qn}@8*D@txZm$4UWAv02(pv_t<;68{;i_PAxxtPDZa4GNkV*!L zJPpfx^ZRKztHvp^d*qPlW2PRdWy3hJUvO&F`fqZFrJs_Vd~`Zm`r)SvGx~{(G+SrR zL3MH-848kftVFcmKA?iOTzql(FIRu^Ly9hkCpi+=rpZ*b(nL?-c#vpA?8i!)LP+m0 z$wakTS98AN$hz}O@Yk;^950yM@s;ur9u$yIR_SvLDXw->jyYn!h?z^mtZf1y;bRsb z&*OW#1!$#&A0(GPlB>u$nsC{c`k<_jkkDScRMz3MV?20dsNlH

)J2p-J9pfJ}p(FpxZ*8-MooQ zl5CG+nN>o)!v~#cNpiAv6n8;h-Ox{>?mTp~PkB=KMO;ZH2T!Jpej7kALo*<3IUy(r zGi|%yH3ciM@@3x6eJVz`+)gUWD0QRL9OL+ELL%8%*EwGzH{g5t_TIAZQ_g!DUnw$r zloK2G(IV`r)_6ziIO8}2)Y0$AvYl^o9wVkYQJ<#v-48s2QFxkb?qoSD zp|GN5z>)jb$#yae*W?rDWO4UZ>&Bw1HpWEI(MqAC_d66n&vfTSB$;`bP@sOewP43G z>EDqlPJap8k~Ks(GPo?QfP^T*(qnTb0Vh~kP1p?cgMSx}cKLi09o`q~S~P)afFZXJ zTBNf1qO!{$%yt!}9+9B=t^}rO5^mi$0_;u^baMA^o9O zU5RxZ*Dzm7o17{rC063HvNY1pq-Kt&^4DhYoz)x4G@0a?hW0nrbiYkma_2qcVxzNv zPa;uzV#vQBS@Q^}VRk9qP_#?@R^KoKC$&DZj~aAPFEh`q(?niBPYxIJ_XwDUZNbL z91|cog*OMKl`v=xu^-n>h8Q>P39l67s!$Mmm4rTs{u1lzBt6lh{hf>D%{OKRQZbDI zZMxE9Q_7fUlb`P)k^8PV8IO5tD*Bz=YcnUb_u$gaAzlP~n`avSRH0`)o777(&xOVH zVCcyr@?|eb4QHq}$1xI~1R4YAAOrIw?%HF0CuKpEMTE|hUqio+aTfu4`PThF4!5fR zQKZg=N<{xo@{+?b>$l4fWLd643+5PNa&^D$)l{PqdK`y0b49i=N*YUQUf9X)83k+h8bcMUD5Cf>?SS=wJU-LNhQxBE#Q2j@H_Omp;I z=$Ipx1(Zv%s8V&&rDo@*wtlFL@}5)NA+&eXcyLW5ODyJ}b5g8=_VvuWkyWi7_Ykfw z3wMI`2-(Q_Z!~m!ay9rV*q_AaeOEUQ@M-xs8{A-nO~3~8CquaFH5kWZ1<|ry=y<5!{G&~VLA~j=F%UUpFeFl>7IB=q|ttZ z7=P#PYs+P3ZjnTNmbf1O{3(wG^j(kuKc?lQ``5-(q0qFj0Hh9^30YzN-ph&dy+Na@ ztSnrdb`LYiW?sA(RE4K5;p>Rcs7j{pG{mqh7fF8l(j}9?YfIu8ygag z<>&=0+1|cOj$w?X9Qj4sU2Y+~_z2|H&&N+B2Bsd6la!2icrj%aIUld+m+qnpU>O`T{^(a6{GthNM@y9U02@ZOFx)Nn`^JK^Q`qu`Ltlv$ zj^}9b=#Do(&iL=b0Hkq(I`!BpiD8@H6UvkuOmxEqys!*i)^*aK6~vx;SLU&NhD#)u z@i(qiy{@$}_*Bjr1NcFri}%~*JEOd6?`MHSMGo~QXz&w`U7Vdj{&dx6783KP9TiQ}Ol=-jt z*-LJt7nl!^s4uq|tj$AaE@T?ZoL)pP(U-sfV%hQE3)(9N?I&E2jGxJ%nKFtt6fIBR z#00CJTr7)t7WgzJ<2;s*Z|l}+yDW8og_eS0?+Ku3M*}7i8X$kjKoioSZ(N6Y11vAi zY66b+_X3JCAOZ!sYm|6Pv4?CPo_vJHm@r?&KdsHH=Y-vqZCrqHi+v z@-^DzN<%|LpgTDglnIURnRVykOv$&`YQzZ7L|>(d(B+4{W|5eZWQFS9V2?;QJ4!02 zO2#7^yXuG52V{SR5c8}Q&fxtZsVcp%bopo1KJgrN>S#XwwNq~-*b&5bRMhQf%`iTxwuW6%}H`;nRmP| z-tHWT!GMvf4|F5#0Ka*1*n3L5EPh^}19Tr}L^I}40@lXe2PlujBg{n--Hz8nik5R+ z$52+g>`$Pu=#+a_``uTYT*_uGjQTFr6nFkmpR6@d+xHCZC5VvP_6Z~u;MLe&jKiv_=n1ZUBu?%TWKw6{?xleqcCiA3NPAEJY zr+B{Hx6OI?e`x_8R#%>xP77GH_c7Q&&E>kxo6W_W0R&u4vG8bzS}&}p=AqeKF5XFq zN=@%|7QDJPYK5gYkxoT&@)k#Ca)7!fTr`t$Opf zos;@7m5+$8QLz;;ri>YUSA*BIjn>rsz)Af!ZD;gzMR0C*by$*?LV2mALZnNQ%H+8u z?*&R{G0`7FWRm>I)L&^ktXo3ORZ<|w4OklN#|627K~jzwk~C=wOOj0KCGR!WZ0AB% z;riuhB!8&9odOmJ??d@@^z6RdQR~w}(CT1TH32mFB;SC&9}_`~x#MP;*dk_dCR%~( zb44v|y_*uPkGL^ofgRm8BgG{}gTI`5T)d!W5H{Q(ERl2$wrEzn**}Z;Ksd@g%8p=} zgElA4!W*lb2K%;#p?@#NQh%=wNo{hJy+Xby+gfNeU6naqxewpN(15lIdi{Xx zO1r1qU8YhjoT{_v@P{JBC(U!R91r6t!RH2j{{WI2oZg`slHa-^)KtfHpAezNlr;Q; z1HV+7WjwbxN-;-1u8NwK(vGOw8x&<9XhE};;)=%KsSssq-xT*pp6`Y}vy%!*`1dsi z*CvoU?*C1NJ zJJ7j>!913`_Y!PBeYnEO_>p-XJXPKCk@W*jxfDbZj@fw1JjR(6bUtRF`R;->I*`0` zfsrYQi^Y$KU*41bA$>Z9xhUPY^WsvGf(>ZgC$V=t1?kSi2c4<3&+e1@-!*Ldf4^^~ z1Q<(t71P3K3Lf-yO)I10t5&F{pD`qa66vc%-&iXUqmPM2*Rf~l5X+QwNXvI75$Y-gYyIPCWVdT7nSyMEQck6TJf|BCyHj*O$x1mX~ z^Hl)UMpB_^*jciTW=%jBlGO4Aq%T@So&lc9>qlJGdLSCI4#W(*Zvd;T z%2vNCx6_^U$ljhG$KRcAlqf|=l9L>h>0b{01xUWbsHyXO^_n4wHAH2Jvz@|HbW}fk z$)*OV2RY=wd&cN9PFz=>{;)P{e97RU>?9^fiBV6$1}Y8ue@uOIT%Yax_tJ8o+*&Q0 ztA#DwwrdN^wr$(CZDSeBwr%6Ny6^k@dp-ZIZJpP3>NwuIK-Fa?7_C2rh2|GWOUUAeMkwwQSoKs4V@iV47`B+GOjpQ~@AL@pHAX*uW7C;c+ zW_j2b8Z9b2e~=vi`1c_j8u9>9t|kPpL2Y&6;LBXHyB-*Su=-`w&4B7^GTrhYK@`Ks z9>PXf?ILSsdi}kAyG+~m$qhtrhXumX@$o?CvMp}z}XZ&Z>6=X2}Vs-eWm zzW7){7i>CQ+Z$>$(zPntz;KHUlcYMS{Kt&X33qOgl;bi|=?E$J5R#&33y~5kHQ$XW zq9D8?DRizcUkH?{nyn#a&~x$yOvbG2wb8rq?GgELQD)RYgJNfZWXfi;p0VsQ2s!uQbHnj zmb_pde;FA~c;08C;`%EE{u0C~n|BNxT)U=R=(?G~Y}gnRfb2-bZ=JE0Z2228=10_! zp*)9DLy|;MW@>*2zgH2saaMNRqQe}ldcHTkS~5{#n7#TpI)a;WNiPtz97e-^PF`b! z(buKV^5x!&;DGjHM82@i!)V_QJU)ov=7Aiy>J0gn^e)>(AF}@k$jr#;b7W(g{JQ4Ki9N|0{5HpdnMm zHWvv!3J`c9jR2_y+@&G`w@E7!*Nj$mm_m;>*A^(Ehsb0t5uNs1!*`;SRxvvv?Z4c# znEWe~R_B$l+y7Ne7gYR62ff3VED$!Ww*TeROLS^hY6JW?Qfe)XLH3_^@MGn^XLppMOKm>NUfjX5{~zGR z69fo64!BT+6o~NpOm?JYWmP!04u2~Afm^LCG|#n?$4l$3B925xg55$53Xk%@ly_8w zj%f7VE32I$CIhfC!Qoj#*$5XLC;9!~JlYx&6R3vY{>sg(*X0GQu8XoZ6fBaaYWvpX z-(4p!ow%tr=)~~^t8*maF5rNXS!tTQ;rDY1w)iR7oo=)kf$~SnhypAEdG_UGYc~l6 zAsG`yG%q)1;H!=IJ$=oe1#xAc^m&NS$M?ulW+#5 zITj@xl7T%L-aUbrh&p|>IK?DWO~KZMCX5%^8ooC*h(^0%?MCzqMI5V(<7V^3Y*$#| z5&&#{B(WRk?9R?)ll@-k$+H{HHwO{ImQQ8d(A~Bl;H(L?whX{sJj1`j?3Sz{gm~b+ z3icOT@1H%8x6xo;sO^78R>Od=out9QkC(+(A86Hcy3TC773eoS&GODW9Kl={O{hAa zZ`1wt*fAknn?hR(^~x7~`01>)s9W8p;$<&(GocpP3W#(5RsE#;0Lx&kEA6*FWHltX zLLrT{5H=+i==u&3FJHTFMQgBtqwR8J&z_KKB4XY|WyNX$?T4aONUvGdZ1YJI@2!BR zsK(vA1Cf;=U&4T98%4+=J(2BFydhO&SJ7*eI^ja%uPPI~Hw|w_fGRHDEh}V21aJlz znB_3{4_`2mVkhoPHz@Yiv*2@Lppfr>?TuH$iBdB-C!8T@GdV9&CJoLMXE}!Gh`;%* zA8Z`NY8-PKy>zfcsBR5xHL$FGa7wb#Lxh~&;2(`>a3d$JY3ygBh^Y~n6$!8ThkhnV zA?zeEXIZk;GQZ^m6aZ8JFB#Z(>-@pdF@=Wf;utqj^_z5 zE{^&)JGzH;y$b?0QUKFfuXJ)EniB*bb#8^-CJCP}1&YY_V9%vMKEvLCtVL(#|DE#>wbY!?zZkGw3h%9M0$!xV5HNZ19%H=96pQexDMKIMK zpw&B2+Npa#sny}P%I}dD=-toe)1$Ah$g@8~-oVe=iqYMsGU1kOC66DOF>V956kiO{ z&G8S6h&V9F>Gr;{XBs^n?B%$B2#5wAV1NztLRyf-nF5`G7;`y$tlryC{c>xao5)@K zQl5bDBiNsT+J%3!FK^NWEN$1@p^r&0a>Y{6&rc>b)Sa%Za;ef80(+uFu2$?^$U+NA zs1x&DB??75XVUToJ`2{@d>%=`tM7;|69u*DWWqd)1GSdDik{N;z_~~Z8Rc)5f-x@> zeQoK+z-O4;%<9LldLJY}I-cQ7FZ zoOSfjX_8au8|?tzIA=CeEEg%UwJw495?!$t{7j&;y-gh=?RWn4^vPUs)eALC&m=OWAauD+cNR%nB#9%uU=Tq6DW`v>pn) zsft-#@QPWH7SWy1)MJLdT;gv(A1|E9_(%7@BN<~A8b?rS*oRiwNBHPt&E~&OVU~R} zaQF;Tyi~Fra-JqJR~*^OphHWGhL;|CZ#j7Eg*`==kKMr?U(ADeMh;&i9o@YK7Xg3` zYFwSjT7HQpcJb4*HQxYE#J5o4$G);WlHos@9#Hx5-G)J3I_qUWg`MU(PFO^BrwU zrbZ_u_kIbaS_a969UWdggya9b*^wk0R+-nOiskpXJpQb=cGaEifH7Rq9bob_=B3lB zzq=2-JyKch*3voca4DZX-G5cXJir>>(FGI?gy69Kz*mb$EetD0JnQD7DwzR>to2OO z=qse{GrOi?q?r#8yR~gLG!=62nV2(efT&BHLS`0cfXxdC#UuxA=0!BgIh-T=X-yJa z2e{nRBIrF^tVmXZ2A;lBh{kBTmK>);@>TcPy|=s{9ueU7`M}LZq_Y*j!*9`{l~kpLR38 zzi)=ep~ncFH?yhk`lF{&pJy+plOJ*7VK-|#qvs3Z_!8D+8Mg;tJggd-Ba(ryFeX6M zBdBc4C0bYi(}Xm_10!!9mbzs71cm`FYjdP4TLUq>;oWOKVB+vstn1O^Yk{wJ6newp z-6Y3WY0B4^GJuFHY|i_oj~mKk@P2|C5FR;iMSqr7*q>B%ZKJ?b6Cki1-lc)v!_!24 zwmB16nL7G-BF2I>pwYZK9_0j6*e2DNv_Eti?j=q*!A@{RrW{r(xamHrL&8KQ2Thmr z`wWA_Vvg9Hmrso7euEax!-AWz!$9zm{lvJgr_<_EKr;cs4+o_oj|TJRUjAN!ofvo_ zUY)LA%eKbPN$51#$Rjq>JqxNvJse;Kjk2hZ!jHPcd;(N@1;`%zSRbxkYtgxwS3kVX#@G@v z{^5#6ZxCgzKdf(1hN$>eMw+3{ujB55IK1^091~^#>nTNQBes6k=wJm7&uM1q6Fj3u z)eFjZ`?uQl4DsAk;PP)qQy7v*f?dgvM2iy-I8FDwQLYDnI9G$aT$HR+Hw06>QkqWl z!I}5}uwW1g9tmS_H*&;=Z8(mLA$*qRyhyh||gbAzA}C(Qu9Zlmb| zoZ)akbJL~gJapj2sch7X!*y_&wZuP}qJRTpBHv*Ain_*l2x-{lz>oR-WOa?;2dOFp zd6Cs`JG`?>j!+$6?(kPc`G`@#k?cHQS-z%`@?-`5UrTfah-rrld{8WHAO{jyoJCy- zh9Szn-b3=AmlSq07jTG4MFz)^YN>5}t!)iv!U|Nc7C(of+kC%Jg?I$k(;l#Nx_GY+ z2cHSKENyMwKGNV21D*y#QQ$m5E(;Mz1E0wftP=8MS*^VK<33+t6{f-|@}Q9?51+ju z&HL<^*y9dk+R_e9hMOEtRgKe~De%4Y^c%Z;1(h^Am-6>-AYD9uN4{KGUftot*-UPO zECTmRBY1=QS~IQ+R^$!(KkZ#5vigY+=})<@UjSvmvVb}!8~HCVE_G|HrTQ*tk&fDD zZRcjTTM@|5Pcfk*zLu4rU4@vGvtIz3EWW}nzrtB=bF4ja}h@!ux`}0gSFGT zRmc~tj;~`}tsg+Y@O_v)8j{5QU0odz)zXccE<69(m3qSOiz>Wj%w#P7-^{n% z8P;R539@yexzL2hzh=Z1l7;hXoYXr!*2ej7t06?8-)k~MP*vr_qSS_1u@-T;= z-38R>LLLWuGCU&nn!944_RM+;>+qgiuzmrY6EVB*a@cC|uYdNRSQm+ZjDy=v1umtA zw{#mvEIT3G3F{o*s-MpUQ)U>TmoD!de=7K(*oswQ5QE5{(`~1tcY;OZtTd_ms&m9T znP?l9Qf?!Uym9|9mLUNrFUwo}Q}LeJ8BuqlL?D%(6~fOa6<5AbkrEe z@Up;2%k+t`kmXj=d4FG2qbT_^QZGj)kbZ)Hsr`|$ru^tKNSsdnk8 z(G^Sme&<@JUwV^zmGKT;S3wt0xq2Qv+tICj)GA!$mRA@E+qFf*(;MuGg8C@x=dnL# z709FL8nx~{f!>xs<&(zohMt2`rbhS6Yq-S{pPTsFO|pRk?&A)`rFg?lS4i2DVG*Yj zMa;N8?1+!4%QSDXp`7p%s=$2wn5@Qkk2-h-zr*{NP8Rxlf&sIC3<;HpK+9*=6QceN z+&tjJ0u70l3dy^SX+QGDX}5&XY1w(Zd^_@o6RHqF8w_syXRr{^O2ARX76IsSjfE}+ z;&En&dgP$*8~cq*pt7u=x8YBANN`DdK38u@z65u$c2sx7dX1&BIjH?Z21*-tq9S!T zkEq>BZg2rttv@dmxC))Q``9QfQ-6szEEML0=faB$R{&KdDGkG<@$JXj-V1D2swk7{fnE(#wGU1Q*8Veyvl@9L{sDJ zOiB*zQ6d_;s*aZhK>&G`$B&USHmh#uVjRi|!Wv$n^+&elOdjHS-&lrS0ioQ(l~7Km zdDrJ=lfO8}(>^xmGpANOP`IN|JM365+2a0-_D{YFr+ZNf5mLc^d@56(?G@XyF80uU z?kI$g$m5a*)SjYIQbUfHX>rh_Yfz#lRCr2czhOD|(a-ux`+7pru718Q>=@9(!1PPy zslorQgrizI!4~M00}~+1nGs;%9KddQ1I>Hv6@Dd6%fSB9pS%s|x(RF5zjhYau9h<6 zCS#Zopg-A=F_PVrLszKJ*%u+VVoZOCXLax z7Vm1#d8ITyp|YIx&%aUT_bg9=_oaI^#F&<=2vG2m{LAI#zQE;0GCT@k{Q*c08ysDxqXm%N1f z^)oiG!ENRQ))X^wlg}Ysb=whTNXZRnmkE-$VUGiY2XIZpogFLr`@1=F>t8X|ajRz8 zj=+hkm9u`^W;T6>s~#N7%s5PFac3a(59&1qaLZb%Rh9el;ve5mSU%Y~_y*9sL3^}s z32P1qixrx`mYsa^a-9olYe4GhU+R!|q&T`EbV&s$4{GEmY(%1*4j?A%D*|Vx%3qg! z+xnnf{N`P=m~rTnzQuMM!f95OhfE1IKK#o7#;WCm-cmLCrC<%Lt0FrT10Mr%p$3!a^CC84r zNI1&rR&HTVK}pFt zz3)2IpAmu&qC9c`dbB3)a(BA>7ZW2lo(%p55oJ11EqJ)3F#DwH8mfq06@BD{ZSdEr zz;;{H!MBnDJ)*^3+q6PaV`|I=MP|?|YM&kH6_)M_n^2!Yombux(~KX1*PF zgHIW#?7Yqc!rLcAa-u1rrS2Wv{|Wm*Htd6kIpKhV!E7Q^Sg+HO< z_xV@o*m(imwnvU5PC`PRRxmm!kbkKNFDBsvMvhR_sKOd)rzZ$)E=m4vJ*dF-Y6#Tm zkeza{;;v%(CyPih@q)d-u-^V;DAEhj6cHpVwh0Q{6yeu7Ka48051sbF1?-x+k#e#)A_izhWyi zVWFPW!}b2)M7YN_qYq5Q%#lBsJ=d&>7)9cI7@$e?za=^d>tL5gHWgr`)>69z!)>)0 z$=?vI5)->YrJ{xCS*;THp!&a(t{2~qdQCSf{_C@Y1YAVU3@;c)i0d+NC2@y86NPezkzv3{4-Yfj+SU8!_2 znNQI_o9C*^+PvR(|BQRyYK8ulT;(4hL8_+_G-f{-khtDYE7%v;{hN51nXPArb)BF^ z#Ah{^xkF|XEls6)HU~%KiFuc35GQ>o{p1ng^gEu?KkH~)FNp{8gj3;sjJpOGGhN+? z5u?b1+c;9FP|imi^a@)eZR9fDuHr@zz2}4B-kpo2ce17LPpf@>(3aPIo|4ORr|}vequ_d{J(b|#M&S$jODB+?>q35XdUwbQ%K3#jJUjW z%F;EpbOmpPKE0SlOt9lqC+Ptj@wCzU(41s!eQvK3HQ$`p0$fI>r01vHXwte;r3vi( zno^8hj{I>{PS@(|mG~r*awT^?^(*M&wgc%)hE?yr6?cBpDz&2W8`b)2f&e^Y*AO`6 zrtXYcX()P8eK*?8d{~Jsbq{@O8$PqHYMTj0$r9XZw5XB{Bl-idxW{h=b*wRq?(o;D z(;@Gb_%3WuiFBK=+nq!)KG>yMoxTw9bQqZBgK(t z=1K`^U-Ex&)Xq`luSP>7?*G#Q{Hmh)YwqQKP@w&KlP%SCh`fmw8rNR|rU^9%-<(g2 z+^m8=s`GW9* zUb6x*O`Pc4lsh*#i*kh(0WpHzS0zHpg*byDad6&246to>1tqO^K?>>9yXuB2jr9w$ zB4O-T(|jb8v9I;kDF}xWeoL!MmvKx;Zd&dpS7C#V+pLcq#b0&^wW12f6Pnj!)a=Z{ zIh3Vy$*jsw3ZQxc(>c`FG5EdFV8^Mcul^q_G)R zLo5RFfg<`M74^|G6TbL%KNehi1#y=~ozaEr`q6)YU*$f6n;#2E1DrqmmW9lpm$$nT z)o4w|qyfi=dKxjxP|k$3dN>xPy91I&{<)V@OryAq)EteD=xdK&6X8cEq!M`D$)pEN z#c6yZBQUd!Sl5nG)GilzwB<>ckjB>YtL~8y8U`kau!S zFD=6sGPD+gqo)EYw|&{B*52Y}zKYZWMg+8^IZe+44RJ7xPRIC{843tuCu=2nJfuXS z!5M>9x#KI^`-wc@VWKE621Kw@ujKO^p>izM8M({XAduYQ23{bq;FCL7q*CWts&PDx^uS}U#>8th zq6DFWul!G5s;?9u7nCRNVzmH?ELFM?SKI$ABjNA7ME~Y!4Ffo~5CQ3PuwGoGEB$Bz zO%Fs0cIP@yve0sTwI+6~s?$Z|zrX`4mY8y{RCeVBR1a z7mk{sMLB%!NNo_FDc=7gzP*%{Wg;0Hb7*p7j2xhF0C6@e*@E=;I$Pn#iK=T_LLbFr z_6y~yK{l%4U9 zf!^>pqhTtWslvRvS!GUotzYGZlSkJW18QVn%gW^w*tpbtunN8l5 zC8bw^QQNu1osixyJ0~0@9b8>N{xe%&!RrYvIXB~Zc2R#iD&+*XI@d10{_q25m);^G zpm}aS)yo|q-Yn)X!sId5bU>)1r$9;Cv58Znu{(*Q-G_Z_+r2c&T(maW)5h4|D;CGu z2ub4>J*g`%DuK?qFv4RP(d?JvO$nQ0aiu|5Eb-lbZZ+=^cdO4z*jq~JsC7yS_DA=| zF2t?djw)69Dwm<0+`@E&MYI%QV@0W@>LyIy}r{kr?ly3&Zu)YJIUh(I=uOs^w2Vo=NtP?M z8Iub1-Q)eq#Yj!{jQ>3PSIvjdM8X+aPcrJORXK0TsscKm3)=X2h+}x~ch@_T&`xmo zU(@s|XoZGJY|~CdYjaewB1)JMTj_A4rEUZSXZ$Y)4tNzs-^s0A1`cJtgOf^G2fTg$ zSNpmHlV`$B<3u9V*$$IS0;`x0-$L!eQ{zIQEyc#86zLgET_DG9AaPPTvE|Hx2*o5p4p_o+^r^Z`6!#VRBGt-AU%RHJ%6K}v&=oQ#h34W#4_5q?zL?vtC!c-3{1aVQpxDc=IZqFH) z6eSVNoz1}_$r+#?Hiq2H#P=w`#GR#T9+`|zN|vQ~%aZemyaoIR`A@D~^yWXJMQ9sK z_nw#>an~27C!FQY5{a~_Dd&-J0*!Dtx8gCma-#Bp=Ev8Qc>6EfZ3(tv=>r2l3KBkG zRLMgZF$E}{m~1ALn3LZnOEz&>LAEjjXKKUxar!>ws?9$hrNU4}sI|t42@tod ziw%E1Gq1kNtRecdZNCPFVJY9guc`aF z4Fpy{$Nc)n%1&cI99gfQj?$n~>i55Sw?p~ve1?wN%r2AHBHwsgg5~E8|6zen!xcAQ zohOW{F@~*y*m)oI$M)gH%4$$0vKx2b1W|+tuUN5!U8Hjzi!;c?FI9#z^!_zKuOd$d zEgGOp0y5NG%_jvV{U>kS7eR1#>6ayEx!`Tr+f*76jg-js7_xmOJ0`|-{aq@J;#!bq zI(g~mKVRm`R$m<&LRl!Q1=4~X^Oz$+#cTt^O$Qn~k_1$Ig2C#+Mg6;-u{lP@I%;zs zZf{;3uSFykO$5wW*)}Dfm9fG*96LYF=a@=-NcY}+EN(*+k`1GJjW;Di0!alghL5-= zNJMt7QkHC-aBf_W9!I}XeiAxVZ@#7b=w>jd%g)eDkBu~UjvL6^cPwHj&FgnD@a@5V zn{$H@^fMZu!W4n;GfHw6eE+}+^Qrbx5^Wg7R#GEPr!b<6SY)X9Yy5N3o&2#lNH;KX zrkaD0dtYlmgqoagXXp{_Zu~=0)ed39c4aKSn{-F>E3qGA%GlQvMWhayq7rBB1H_v! zFdMC5fCu?-nXZI2m&35{1>(F%>G`~rfHm7%;)u-9Jm|s$Dw^i4$G_~3xLp-jzfD{E zXR_DB-<#I(mLU`CAMfR^K#Q**xRXg}Q^LW6OTy{x2wfJ|r^@6X{|P`mp9tT1h(T-K zicajZb`eY74J>!Lm_tDf5*Pu#gwoF%4EgvO&@tg7v2t@n8iVaRXS8{5u20dl>;0G+ zNt)GaCxySiR9<(hU299qI~A9q8WLSzA7@^o#9<}L#W3d9u5jrD<;zXYwtc=`Te3WU zH7df7#(h_s%9a)f*Sx4nEOfD>>IDyelh!w6||c$ z7BLFP(KH|0WV9(3-XAW13@pHu7u6koOhiK&Gc%-J1`TUmp;O0tA}&G^Ltp+$jRi%? zqO>$CSw|@*DR$0~W6OBl;DV-T&OnMcIeocsT$}vWZw+bu>rz_aF@RTO%Ip|Z4h+)y z+(vlM^7%F`6ZwoywwlNIL8Gf#;c^S>$}UJlRFb9@H#Pm&`}+RV!~^P`r>suP~Btb088_1hJ1d!e1)pyYpBHwMlHqc01eY& z$tqKS7oo|yaeKhDu$y2~xwi#Qql(-D%3lfJ8V*D4KXtE@rd^CCcVb~)b$*e(7c;r_ zO-@1gm$%ob1M18R!oi|cj-=#2Ts3u6mXGl$wR?1W|H(i=T6ySIoX z^2tiyOy6o0O2tfhFwt$}wD1l~E@L7_yp8&A;Q0po380=@%R6$a9}o}(tu^rQr(F;~ zDhjbvf6G$(-Y#qMsa3%(ayofdR4*O3<&D`L%Q;S*dOvSR_({|!>Rh9b>88AHk8f2< z(s0T}Ab4(2`^#9)N{GfN)h#uioPbPR^!P<0CtBpS7A>MEh|qrBeV7~KU{L%xryhra z6*KL`)yrv}qu9UhIb?=vUK{C|P(`-Tr&OXiGdc-QfQv_3Q;s{s!&WuRJO0k8;B@$F zt(f*`?X*^{G5wl6Xsqk1Fg_tbtBRwpd+u|F$<5to)$9>IZ4Cy)5in|P<jI^5Tg#u=J6c<1`KtUP4vsyQ;C4NGUFKlMQ3wbSa&2=k?8XLTFBCx3%PFZGgdSImxwaS0w*QNL<;`d=RDrxr*5zVXNosTVD z-E?lQeZ!m&p`AN@jx-HrmeH44|9G}Uj=mOJ?2odre`k?&C#B3ovY7*$KM*@{5689Z z`TTXd&@qxesF4CC)41&#;oUqCGqbG-eZ|t#=3^nd1jJn-IxSYmt48ydb}cdz(s)1@ z8zA>1xBv&|bl`nA1A%hE@3TvV8}7+)i}DGLs59{s6rPu(-4u63kB#NF@1OeuAn3~0 zOgCTpM6uS7XW4lpLRe-)A|v8ocmmKrNAM$hy7&&8FbBX;?Ye?Ft!s@ye82G=8&vI* z+cv*;>x;D(tdIXZt>K8psA82^#f+m0Z57h*zNZ9H7)|~oagySxZDE% zIwWZww^rc`nTuVmrcdb;MM{P@efQea88~dX_K#aFa3f3opZf4P1eo{Rz8K*eAO>;U zE^zIIlq5Bu4_zoLuWJg+qh)~b*<)?-%+sL{$hJvUtL?7qp}-&)Z>{`3m9VMU2pe6` z33WM3oGXw&-jDVbd;!r5Ittcu%Zg(g@zy`Psqur3{nFcUKs!rD#(lJ&>0?;jUxGJfqB6<>|IC?=7n1Br zC20*n>s1evyGW+EpV+n~iJr?uu6cH|SfA3W=&+4&4Nwo$K2nhpdr3l<|ZPMjrW*Z!# z&c86rh|-=7yJ-QMSW~XN;=4BigFZw2VZSucC;mQlUH$mz1qkwfZZTJD%x8&86xASm z$dc=quJxCGP7oR3 zDlB&?xrKqbMygCCEAkgMd(BYP%b4X+g9;Jx6BhUsHNVFE202JFoEz`DoGz{{jD=XEJ;I)QhiJi`)u_~GDRE$iYp<@e=6nV5|(b2iqH8=HcFbiU^kWclPEdhQQQndrsgmOXk{q3XRaB zJ@cPM-j=4i6RR^=*|uWem_X*uRk`q`55k)xk-dJiZbzz5#-pOs%YBiPh%gG3!m>VqB+s?^>wXcYZkmPFmPa zkSk~#+9Uy>TRI*yJS}?YPZ%|XphcZr>1OCQj=3YB1=hr=iLab?%RXx9%ax(-zu5p6 ztfwyWrQ24y_ZeNaNdyLgGeyAZ4P39SlNC<>zaRsG^SZxI!)Y1 zCMpL>#WSQ__2_~BQ0j0KQ6$dk<^XZy(ABEcwBokC*3QWm!^LghzR`P#=QArKA>5!2 zH-%^4p_p!0)+3OGsr)ik;#xd7Ec_&vEm9SBmV+Mx`gx=Hqin^xpMX<&y6DQ2e8SW-;bx^5k;-yot& z2}Wg*?kW^1qz6m z0>GK?KCAk7NWX_3dwDsnmH8$ygVYt4qaWkn+kD&zp0zdU?o;tye1WLXtj7gpB0}G} zSkt3WELp%6_6=innkuKLv^6Y17S|>c(~>Ezx`X0TJ#NS{PlnZ~b8fe@ib{-NW()?u zcv%@#-lwhK=|7;M5IE)%#(g!BMcQ3H{9=uOe4IkrkOwj+9TJlcEY=nM@yi6nGnu=P4rP28M&-GC)S<~0MDAmV4KW{QvV8G9@JTX zFK24_3Un>Ff^~5rFDG-)NK|6es&>e-)em0A5 z>;t!^P0+;au8`3eXX0JvU>+>mOMU`o#19~8KD;p`fs@{stie+9ml4(|iM`!d#hlgGgZ z@weoWcPM8n>3hgd7AHKbf2X;47jh(h&kvH7c{tdD$i}ut$j~5JY`pR14|rXIk+mlf zS|ZEN7)J3%SLg5IyhsMfLsz#C=G=7>n|P1foD{J>qRXL#qiriFUh}ysBsLDPa4RQs zcWN4&S8*;qUF`kqms2GZg_fH4WuvU8FrxLEQx(u}z6%AIqDQ}@xt8#t3TR#5PKeI| z8E@_bwBjig(6EgfJ)HOchw+B;IKN+nZ3FsO?xH_U2n__K(Tmi`zT<+SJ#2rxU4sRZ zP*V!T+dTvb%mFMDT6Y8@wUM?tI z*H9qnP9n62&+zVBEh7+fLYR5PSMo>1$yty>D&EoAARj{~M*zXS2@v0Hnw1TO;Q(Xi zndQx7p|U#Lr?3L$pf(UtTh)x^Tal^ac?DEOK=C&_JhLu}uB$c3+Ptor}15*^6Y_iJcu z4I*KPVCS!DVDj7tz~;=dbApnH_H1kqea48${Dd8YM)>fm|J(!#n{ZbUGJRum`Q?80 zO~B=OUO3Wm7eA2ufqaoWwT{w3{WFd$Uu2`3)Oq6#)xzKS^5uBo5UKtIh^-B(5I1b! zokyf7fl{Gfe{7nmt=}4!&#oj}yqvgcNOl_K}B(Rla))t9AHB^o6pg-8{*g0W<`R6 ztzaCSz`z?3!MRUU5U$4QluP#}ChFFAHg;G{w*B2>}(rH#BG4RXv+#rp^S3!3~G}KlR;pLzQewCn<8;6*0dcdsx zCbB#jRNSw5CKZI+4iSHC1ft=Hkwv6L{qwJ!`6PaoGJR2}`VQ`t{y#i9{hddUW?HeM zq0zzQVu0jC+O$>PM~TJE9iC`2 zeHuv#6*hnzA>dhUG$}Xrr^{RF?nhjO1o?PX{dbh>K>pu=Vhk_adXSk5!cVvHx||3L zhYNJhM>V>q5v02imwdsYrtA;&IoUU`{@0Xu!c`N`?x ze+TUUmw&HE`TpAEmZEN=gOb-iMO&OzB_?)}fADU8195=cu<3yX{R|*5cw%Z9*-7fa zyjSqhtQ7ltzf(P&$N}@d3ZcQ7+m}Ii#I!D(cu&P#+wca`fCsAlj8P7OVVz)geQlpMo;LeSS;z6~ z@N)`^55q6W+JEYV?A_TpDDujcrFF%^Y;$%OmcWH)c9PZa>E(n#?ShHkz=iSXLr9?A z{AXPXc+BpjW66CB)nGo~FL38cAK z?btM>reTj(D&Y!nUtgUQF4sJ+&(cNB+M(D2!V$=mBw0A(G~#BKty<*t40g86Ft$Hr@Z-AwL(?6q#liSDj= z5^Rj}HoR>^cHRKq-0y~&;uQy^%2ntSLn^}c;neweiO)dm>j<14!u;!0Mp*o|*iwo+ zM`X@-N7{oyDP?|_mT5jZRJDFAcJH1MUAw~Wh;i?t`uc^Mx+2I%7cw=>Gxa`KBOGyU z|2TG%vxB{ibcLW%M3@#eLGDbuirsa2T{58DTT~;U`-ClqbZr47d-YjGV!lGk;=F8f z%qS#yl?gW5JiYdNheg_5XT<}()NmB}`;|KcTOfmgA9$rAOs9gwMpI2=m0AG}1k zw^S_}{U5+5IqnO)a8%ij5DkUTH|iOJaW#2)C5mYu`?T@?wHu4;8@2xUAw)n;oJq!Y z2YQfhTs;lmkTS6nd^Q_YJ^GnF@4=U1k?ABmyqrX9b5N&`stn39eRS^*Dnf3#Pz)&n z9p7^~?$94G`TlYk_4gHbp&2{VtOueY)i-N#oqhq=xuqtgIag4C0XMF9V=wxsAPv6i zm%}*Q^rE&SqT!@(;Hqd~PF>`7{t(nRzN{vQ&_vd>P?{N9UP=fzaI*uKR9&H`-H(-K z=YQ)h@_x=66XgW5<-a+&kau_6Q0EN-iEtq8tOjkRa6tU*jZMBwEoTdL+4MFvx4xVs z9^49ObRi@|4za?=hMuBUtX)21A%%>;LGyBXc^`)NIKlLi_VN38G*h6?v*<7Y0$)2V zxr5XnC1rl&O|L@9E)&uql-b)9_@-y;ebBD}Ou&8zCg6Zebg?NHsJgJUISJ10?nz>f zT|moiaD*S}cv9H+&^CPS>3KSw8c!_!-iwrnb;j37>r6jiol{WBRz;pY#d>9hHHu`L zoYDOrhshtK$(6qfqrEmqLvEEW!+d`o>yso7I zA86jbz@e&7{vlHpm&ZB_72SgK@_~&8mqPzhwMTZM!8=;O)=R%pGFX7AKC$BZeD&oS z4#1Wyo0tvq>dt>OyxvEyU@n*!`Ltv0KXsvWk(3k5*mwPi#!vSxT$rITkaBO_4mm0f z)Ovy^S!pq}az2|}Apio)f9DNcy&pz61n1w`Q0%oF;c`btagGrY#RB-MG>(7oeN-P7 z*MWqDlzUxUb?ao8)d%5nb2wwk$jBJU(FFQs>zu*CPzdru9k-Pt4wuhY&DFJF3JB&= zj$SGY@a8SRi=b5z5VT#bdq~@pOPG@llF$#PkaYY%y52gj>F;|SS5Qz9K~WHtPy_^| zOBzH)I+Y$scf+U+5fzja1f-FamTm^pF>1gVjPBUTG1-XUz*oF|{l1@n_IU1jw|mby z_w+sYUi`_174~GRH`x~=Knv(g5(Q4YTIH`duCB~)iBdkNgDqJ%-nSc3DzbG0OmRRC zCk=LH%2A6{ePej-nrx%Ps_GqT<{(y2lzdeWo{aUqd=eRab(=MqaN!mVS-3fs(^-My zzEW`59hSearISHr${@Y3Drye*44@Yw4!QZm_*=HU@+#n-e}Ud3wnp#gtxSz~w^F|F zoy6I3y>OcqQ$w$bS{;t?&-Xi}d*WuwUzeIKj(2&){v5sFq9w!m&ZyEsk0+nu12j8ePRt9BDGV_42mR>$|>aZq#smiSN6Jgn7=}ZmiWTs^sk~_m~ z(3g0zY})dj8q9(;g==}k*T+O{f`Fq6uOO~g%r2E7-NPktTu{B}yjz{5H_~MA78EwJ zcW*dpk?aXYhW;pfgUADzP-+^+jI?U6!ltlUk(2&>!>DiQ__4D#V`bk8jUKSp{dw%6 zMxbDvJ7D>}vzmUXV|>zt zl@eL5&!My;)NGNm?23uY0%za8Nq3cmQJ(uJRUl7=teeSSH$Ghoebcz+H#UIQD3-|4 ztQxvDS2>=Wb#?V_4L`_)HRVIV;cm1q)b3zH=c512cY}qM4?&sVmQ6fd^4V$mhQ5Rk zWS*Cy@SreBKnjO%uPMa8Thk`3X!BXz zKcHg#LJV|_iGAH~T%ul%F48KU{r-V|u{nbzE6Yjjp!f~k^tmzl7sFrmQQLB}2?>%C& z^>T9e)lY!iM#a9yf|fN8z^etpjYZaOMOUpy#JP@OAAyuWeaNe%EE7}kTQzNBwjgy44z1Ye?M6SuR+C3sRIA#1xlx# zOH61Tpp!Sc6`y5zjl{L~8Y5*lw|fF$(`S;j0#Ol{>@wxkDKu1p*p{WQG~7aCcG@-t zGTO0@#X4Dlk>nwMbOvCZ-QP*~`bZpi`&c7*ACj77p?QA2m9rWH-h^A^aCi_25BzFx_ElBQ`y)g3@Ze(l@LCw`QkBKZ|wgji0D1uRe1m6 z4#hDoLw+O2$*hf2rs3}J`baHY4a~oHrE#<|5#svoun`BTIOL4Ue-H!pw|!N>ZM6Sb z*g-bg{wJeALDeH8Ebv#P03O2j%)-^lCEm+Ynj5dD>W??=SLs961sP6qR}1NxL6vB> z@g5_av&Ikviq_^;zKbwVa%Ol|$g&L++_7u4Ky@w6W^Am_%d<8KabmP9%C6XQ{lS;d zD@9Hl?s?cG1<>;sBRku05rrP6oqDR`nNf05BUXA(2WB@Y2$aHZQ{p`0RBE$snX6ry zt}X18mxq5{SJ}T>$|);ZNJXRine)U6Z&0aktiP_jY)Up{IOLH`Gbg!P*+tP6jF`XA z-7Hd@Ai`~%vm8cNY?sL7$PJz=d%W~gO(@dAuIl95vd2m{N-g?|FGGBr#>WCWpZB@& zx_fB89_>69A#TL+;{AN$g}<+prnC-FnyS-aJrU-TXg9k<&x)n5k^G+6JAiu;X$-<_r(MlS{s_F6%k!dzu8zbL>G{KuR6onlBHs1+o}p%P zD?xETp@m3uB~p#FZ2>GP#yTEDrDa#Mj*xZn#jAmkAWR{WBuCJ(xQ@z)SY?|Q3Bm30cHYIfeve17j& z@vT^P9gVzXzp|Sn-xFgHP5@rHQ58Sd*%FQo0T~X)R7l0j4Wb|C|GLu9>&gmESc|cw z&>XODGc}V zithc@9zTx2(p0K2@vbdj=3jxKN+0mNM2W_r55Jumf+LsuV5yf=PwKV~zi=SVB{iW9gQY}NS;kKL{F z^{O-9OjKZO>bSO=5E3Ex%svXbE&)d?bm@ytNFOXJX*ij#zV)T+g+65tPP{xKvF?Vt zBi|is%zWatlVFM%uY1q$Du58~Ph?$Zm5ke>wBQ*1LKpX(2B%ideCMyLxMilu>Mi&= zHoxw?@JbQl9F2^?U&RVJiN9I3Vj~|Dp8WQ$e8R%NELfY<#sSc#wkR}lkrjrf(emH! z7dipagO_bRENFIFzP*MTmC@-QHk&hHy_ci5hJ`L2jqw40eS2YZ2l@_V-|laoq-WiG zS;m&KY9yR%F^ZV9vr!xsuL;SjfH`Fg@1(L)Hdc90O2$0WEgTWw!(ZRQL^1xvIADC% zs_n!@eWziWF3_>dxA6_m_Fm(wj{;r#mrsP7t8<6)EocUPnpbUgj;A5!b3gm<*REHo zsnrmydn3g+3I_(>3XjQ(H+QtB^~8iEpZ7U>=qoNW4K|C{yOtJA)JvI`8R|MK6MIUZ zNE!>@bn5-J+-;@V~*BP%yln|Q?28^Pm^vRXPfz;*L5tG=A0In7kA}oyL|dbo>2Bj(@|cJ&CSPqVu&_oX7Q1l2TAsC$rabdu;OU@ljP`d-;^Vf&wV4C6|3;J26|bf z_V5+^RA|{$q=%UgKEdPQ170bnrSCes!4!PI%f8$L%C9GWEQP4`^!_=?LGKx%ZSQOJ z6E#M-^Y!dT^DAmieF%M=X0Y?Y9_$1AE7U|W<7EkGiH=Q3UP{YLC zrlPHS?HbWcUwqHuEcVo?nqbP17EkfVg~rbtSo4hE@qkXUw~F~=URSr7Qy=>-g?!_g z#<57eTDK^ja+LVRZ@l~fc4w6nHPWMV3))_m*qD9vE0ji=39PZALwC+6Yj<8VUsJ`n zWr!#7y0<~Y5rQK`!!OV>}gQvohn(Z(#$pT4jQm(OIFAeGKTyDSw1x`jvraU3O2_* zRS6WTQU&wgeD`TYkXLAUI3kSZ*Z#PUK(A%Iez{#$K^*So`o~yBXAvOO`ltr{JSSPG zNW07&BOnM|3|YRU#m^&0Mhy_zzUJ+8-0`~d zwNYvmbwy&g94s{^t`Wq%BpL#QjCvoMN(GYXoNz;m544m4t}DA)sq)ZIAv#k{PR>e3 zf|0rN16OCY4b(TCaNdo$76WGfeUzvzdwk-x#{0RscCjVgTe&CUtB{P^>KyPAFwana zEz$Q|Drq9_$4UjG{QC&soEkVwt@P^l-ZbZOW^nO$JIoSc&@t%!{>dpo5k({BoR}5 zLgx7i{A3>!EDcvHoJc{<*y;!r0YEwJxn?)Yscty}?uju!0@`;+U1YQoO_EW1PafnW5fZ zFIy8!YWfKCL&5F-q{XYE~69OkxT(-IXq}B*J;I94xH=@Sczs55ee-LWE zwwYuh56T|-ErgP^h0(zu>~C{Yv%mk`;U#L#sP_Hff@ohm8}VjR!`hBhfZLNimq4^qh@N1!b7#^wFx z)fp1fPuw?e=iBTrN=w&>IA*7!^OwuNe}6b@Zo!zgRhqGVu*BO=_Juc=&X##LOiu-xfWeZ~(o0|af|2*Rz;{J zMokE8sMWH|YRIYIem`C~9q%^O`sE7YY~z&ecr%`eJG#LAk^W_WW#NL@JD)~2e4+lT zKD>Ja9r>%K{ybls{P>~k;{(r{_QEK0SUBV|Xdy;H6v2-RNPNJt|)omkf^oG?MwG?3$)y z)u?9C6ayF-?QjR)+0LkpDl6_dn9A{kacgExOf*(m$^DgicqnlmR53))qzup7M*C#s ziItuBvNwG4BjlInNgnL=me7{=J21Ws96X?I6zvi(0Hdwx4RAA1qcjLry*PD`JDRV}SJF`6Ro&Ii zAa3t2@ zY>@2=aKFw>uU3$qMe7qPCA}uLQHN!BZ3|fbP`WX!T}%%g=C>OYe8Tqp;fm&BvID`XZ&Dc=#mmfAS7#rXK);88(}rrZ&_n zO>%fVN%hCS?zvva_=tiLWY8M9VUmB$KV={cz*g(1bWtBw^-Lc2J;QcYtRwsXW*{qR zNOqik$}J=D9uX~stEOiViDqRVTC=jU%bYjl+pU*~ zS%{E3RQj3${E(~7zb}f*zwg?Syt43{=~vnyom|iMtL$1V0{Q1~^rRDiC2wSYT-<on^#qMJ?5b_DQ-u-meDcA4OJ2TR%hd^8mJr%<=FJ>dHNh( z!8pmSKP@zpjVRFfAINtsfZJ`+72GuiRE+A3CT9l(F&nR6zh2#|-JW)ofMEm4D`KvU zq-Q%pR1W}~@P#O>W2iy&$f|iat&^`M$3NLDv)mR!`ho7VXsv0BqNtDVX7Ef_gV??{ zRjAa7f3r2}2k40SDb0kdi0_@FR3`RKz} zTpfin20S6N>5Sp$&lj>JCe_FM(ee8Szm4EX0&IT^>2p)VDm^|0C? zQfps&Ud0y1PxFAPBrCtcp$*yB%Igpv%H@$cpayGWClB9O``T3VruSc z-R%T-tZ{i&ZiBP;&QrADeX!@>wE+8DgH_$v>wqs7UtI?>zOA-lgPvT2hlxTjxJLU5 z-Mho_QVpcjE%d2CIwox1_38!6PN#q@JK>(3GM&!!#XFe*=%=A?HYzsi`KDmC1jN0_ zWuibSkWBfjlsa!vmtt)5wK>~lsJRX&oTmYIvC|RvX`eaU(WyS|gQbS;tI4?vr6DO1 zV32C8N~shYB}BD_iB1jGT~UCzC;w`v-O2HG7`LJGqlLS@(Ejm^2J3U22>&^e#Ywp- z#3mkmXZ_aP{PYUd>tc+C^r408(>Q(NsyMyNN2#eunr%1oyVmELC+ka1G1!vEjdk0d zhmW zE?&5t=V3%8gr|ecqygr;Dw(FH-IeK$SLrb4?vLKB(C&KUa#9$WkF{)|8O=51eEWpa z+Fh8S;6#W8Gjs&fFtu(~R*gQ-1S2;b&3vzJ_deX*8;$?wDya&S@n_bWn3$~Rx55Ap zXEXxSRS_Hlqg1^h1L7>4Qzuo&Bp1%9unrM&Q_6Xm%hkO_PD1Sjd5dge`15Cq3U|3; zQWEe|I(})~qsUD49}cuRxCB(-@_2y(d|sJ)a;5~aH+4n!-4Tbd^?<=#_g}=CB`SwF z-2mE0#IolClCFDg;w@a;^2;n1aa^-}IvmCifbLElplhGNMtoMR?K0ONItMhTN^=*V zJQjO$?-eB*Er)pUIveHe$O#{7#!$oK2oiz4p3=HTn*x_rFm7>xaQu3ii8BB0Fi$Sd zVkF;YbgG2)Hlj1MCgt)uTL)x`pTS?`L_tw)568M$`i?3>l zs~1kRcl){F^lDrxzVL7?$B*P=aOIAvu=-N9RGe+F1ZJ^k)(~EE2<-{MROT*yJ z6$Qa0M)_azAEiFRJ^667=c%pkX^vga+P+pNPv6fu;MG2r(=CMJ-n%>T?trnY>Ence zUvgMEGI+GD_o|z{p-UUrXXJR9gsr62SmH(p^VF~Bp?o)sNb$DK!;Rvf3 znz!l-6k5q`S~B(r}zkYv8?QEQN}{a9)Z_Jt+|yjCy}ueq3cK!8So~({}UV zsF<*|lzbA2kg8k0jbUjM-jZWAqeu!|0{*Q|nShZ;V$U@RgFb`^ZrAB#!Wsga#SA=S zylL|TevLugY*6z$ZtK}}(L)Zcn?g_Pl!x#Q#&OBYq9nyjw`4|9*_bdY`cZA^vDB_MwSEFlWS2tly6>fZ$uVQx{VNY0y0YmKJ$snM*`imr?B2{?w@NeG z=wv)s^Wv*cQH9ODs z_B*15tq~!zbYGPbOZ&Q%CO^mT*c7$WsJbWH?@q<*YGtwx23YVTkCH3YC=TQ&P-Lk_ zyMqop&2QCyCbsrSDSJEtkjboCc}KD9q@U2|+YF$7=%{_HhTBH^>B4g)0-bSd`z0-^ zyN0(1)D$k_=?2a<{-wDb5B9*cN4s^ixz?y{7t4EfUQT|kiP-+0Wu<^0*@dR9a?;d#4ybhkGxLgH*wHpA=z3A#74vyTc{dQ& z`%^c@E=R)7_jP`pnB+#yP^BepiqO(0b;LQ)dxNq!%xE4S;HS>au$Ti{yR`lL%G=Zl ziWa4{JC!6RNlR7!;!0gcM6!X?o=`<;Gnh#~RR_87B5}RWLo7@_-C-kC7BsoPN`(#^ zgL1-#5i)jT;R3~^n=@DVdGx9t-PCnW&`EwKUIyP>pDtP29goOZNa;VJKd#UbeIEq< zgw~h(X;Wg$F#)QHSgm;RH948aN%nz}^9$p9;UnQre(#WOtt!_cjriRT^Rb979!Edr z3^}>QyWM(vuByu#ESd#vbI3q1=W*E14$!IXF19epkl84aK6S-a35WHhHzBvI{5~m| zp1N&0D{iFTlfU{lkjLHU;r%UqUlQSvw>T8MT1W3wCaEndO%D@vEv&as!lA#2e_+@* z9KJDh33o80YzwX4v)vE8j1~UKKjC5Ed#_YCBz-mTZiAyo$zzZ;;ndj)f%kzv{$)g;mDi3#oK*cAsFakrO@)Sz*ocK7Ze zJ*o*gT-N@cv0aqjj%4ey(;mi8J#E74$l)`AQ@oPJ8Nh@UPa}FYLSjZ0K5%EwEsQX?xN{%8Fad>e65aDGfDQr0XA3CrbR{*k> zz@mV+Qv^ z7MV|09pZecyvihDQH8RYJXPqGW9p8xSM;tvmb~HO%*YK%N^3o<2z3Xvmx@xZwZg5O zHjbRvaZ>{c!w#N~^y6ILu_EI0MoQRFF^sfEJQCcGmzKJFQ$uW7noaVdkMx%9e#>*; zJ6t(w7PpX4@WkQ>_er=H-fUrG+jV_`w54vPL4Px!#rv_HafUwiGXcn?NIYrQ&E@Za2lp}ril!w!PakUC^};g=KVIK zH$KFOU5)xgIvus)XdiKo=+Xn0fw^|}QXe@NB*=1dL^e-{BT*Y-A9{|cneqq3e(k$p zq;TW1I@Mi{=pm+l1}sH<&98~A#*$Zj@Hc6`viE!knea*(+`{f2;+yBb`KBcxd)xT{^2q;9rJVSs^&ixItnB-?PiHg9oO%HJCHj?bus+Z|S z(+cG`v1BiXH)`|t_w@l!%e<4R^DPUK$&0cS%gsMlrcF(kXNJV;xm4|AC@77r`C^6` z_aOm*0NsWlU0wQBH1j8(_k#^?NbZNR z|8STzF7=+ejW|3~DqR#Q<0xha1EOAKYjUweg2q8^vtNtkWwnoCj=o_B%Z>t-> zj$H~!x)mHFO3SknYh6lrIK=Z=6;=R%qCG_L6d)+au(BZ`vl=!IxX|0sVEJ51c=zJq z`}_^+{A`=_npS)RNt^pKCm>A!6P^1APcJ&VLM!L^*y1}wUfqR2$N6Laf&gAlzPHy)tR0WJCASZEMSsAUbw{TlXf$*0&gZ&0?qYFbtmLt zDPg+9ssh-(mwvv3r6RTQ$`kq%0uBvt#Sqk8_X3{23alp*lWamOYL#b~`J%M{nxwv!OhK3k%qC*GhIS#utcbe+Pj%h?Ad+_xV#Jb;|}sIQ3=oGMS|mj|3&EvfyuSn}htf?J!`Q_NsBDI6VGj z%BmxF7GG%zS|{cO!lLMczE;^w9XaT?K8dUiv$VA{M)w+$hf0GEC9i`XBq(tMmisPf zHLzvvo+a8Ff6Xh^nEdp(zC4ys?W%x|>uV&K!f!7aw=ecGMz{Hc*bTs`_uE2P6}E?* z%qMV*OU88`r4H+sR5O#cpc8(JtWfIGVs*Jsc_VSI*@jOJvXisy8-yP+6Y6U8Zdrp{ zZt9is9TR^?0D+OVx4K%PYaeG#E@f&kjm{|miluO(1<4Kh6SKH`$-~;9m$tXVo?j&lvo0{d~~>*K!RxpK%4yHq-L@9AMC&r-gM?3re{+NXC_1sH{R zlN6upDl9Ek6d%1O>@iwJp#9@hY8#HSUi!y=zuMmVUTdNIWa8LUuQ--1Wb~8A_$PAC z#g*3$X0cq(m0CSqtgr9C@+M~6`n8&#`z>akuzsRsSuYlY1_{Yz#6Q=s8UhQA6=61R zT^e+;YiNSNi-S8&fc@(pTGJIS(8(vwsrl0NLZ%xTYM^_2@dHaQBZLtb&o}hd=?IsT z#7+#`ag`mXP{m--eyT!R1~fsnX5Gv?qXm4kMM8MLD7~3S$*BsygEBzsIyV#X;G$(! zn%I!4+i4)_va(dzHf>qpLXMy`yY_PyKnTtEkgC+^Mc<{bJ@?V^KGQ?bmb*K?huHh8 zjyNU|!Vsw-&yROOUi>Bf2D7CXWE&yLVmtN0qym5sV#}^ezO_;hc5HDU-bk$sifHWV zD`l^OOq3(yvOehTsckZ&@n2(^hw$HF8&S^y@r^iM6~;UkTOXwEt_$hf(HBLR@GQM8 zH3sQ1^L7Z8M(uO|lY@A(`_!RIf^mlRee|UNnA#~VWAcxeR@SFwseH&)+-bA-eBV~H zmy~GfUla^B$bQi((fsB1iB~qp%l!kv%>JTe3SKAvs6yH4a^A>fqxyA@JoJgc&5lIr z#`(fXv=D|3;-_$0Toi1Ku}6sR;r4YxAaPh~zq=*rwT_PoiwUfp-(PD3kD#3E#zz=X z_fLK4mo&_!I=N~!a(43MyeXDML;LH;V4My-PMJE((yKOc)NZ7<5S=Y6cBE;~<2*n0 zTp)MP!yDR^*x;9PnF$_6wlB`PSb7`a_JcUFrB0etKfKAtGg6~RXC;4aJ7LscatVW~ zP->u_jW6>Hux2MJmZ@g6n|yA(4#|Uh@6>gTNv%Q!?t8+5uHt?)PYRmV#?tv_%nZfU4!U={J!A6i`ZV zcl}&pfUJYhu}`QNk7+{f!$iyG?0keQtN2z{N6l`T`~oz!k~F$v^gBt@T*c*G>8Jpz zD$;>)+c6CW30w8r5Axp)JZo`tRIa{?g_xk zEaKGJ!zgIL9FW3efjMmLs_;rrNT=PIl(gn`@N6)IoVMM@@h`J|SxiYtd0L1#PpBe8AKJ8KloM_2DH z3B&MV6mf+mJ;Yw9OzdHfa%G{^)d3$QU8aYK?`KYFx30Ez!{v?Tl*|c>oimlVu`Z?M z(K-^waRSN2`r-Xt1#g3e>0J< zz;<7vtDeLs?lOay5cKD&G+J@6l%h)zwY%{uC(H1JC*CaUM(0RVVIP9ixzJ6j-8^N@ ze?p~9J3?6Y!K4~7OgjEi=ccmJD38jFCvMIu{Vfnnf1oz_XxUu|FIKV+Uvv53cM>bX zi&<5FqeY`sO4xEk7h+V^Z#VnMqjCHK&t8a2sY>93mRc!Vz#+8nAsV@NyJr4Jr%$*~ zs5dA0vw(7YI=p7!QAJRAqmZ`TxD6nH8*3<>XdWg9Sfvx;-Pp@IZ$i#Jp_lH7QV>@^ zoU3YCc)gofIVs4zc7GSQ3lJBMBjl#Pw5`W>0+lS|(?%lF$&3*c_mQ;P`b)C(hk{2N zlP)VB8hVwdd7^aTB9a))d8OL%4_tS$<_C&q`T5R$@ai=!};0)a2G)Xt<8v%sVC0+gC;n$4kT^K zk2I14pNK}@xhXoQK~ZSBr7;|lX&&q zzkU}5UYF^&N(FK&{F+f$qGVA(OB6(W#CnI}_vXv_s*8AW(<&hj@J^1`}vM|v{ zo?)8Oh5!qa`h6H-XLfik%N$0r7`69u>b2$IeJFp}t)~NNvJ_O+gECf?N)#(!r}!~M zLS_#EmESEN0uW2cubeUBZmG7^xFLPd+zU65U?3wnFllVzp>$3!T$t{KDoLqewBTqSw(neABA+$&_*e~NXN2Xw5?y$prP%uIifi-l@u{>| z#zj?Vs)m|MV5VGGvLNu?10=loJwG}8>v$YJUHNAic8wK}UTBbdU-zLn_U-^2Gl>ey zDzj*+&NmG=2)tKoJ)y^HLd8ioW#p!1=ehP)Ru$I2!?HPj`$}(YCc+HvdP$%9-YXcY z*a4`~yP;ngHx615o%K~_ZA8|g%p{uz`iGbOd%p;0$Cgf4T`eyv(8|}hOJtbbii{d~ zjy$1faD?VXPhZWIV0l1Bb%_V^HXrsq(SkO*j^3Cq8 zH2iuW&G6}9Nrq5K)91IBk{$qV=vdLMco43mrB&G0;6DlKU9ON{IscKciiW-xeS~_M zV-!|o>kIH26shxvjzhq6$+KqD?pwE5*K6FY1<1eKeWb!dgwW5fq8j(;2CGck`|%Im z8DaPW9hk3L#m-9=E+q9(7NUf&BU_l~YtXY@okv%>o**(T*tD#$Uj6>S3eS3}N6ziG zCy5KdGhQko@RA!hKZcO#M0@uU8)7RKE4-{~9a9KC8jFJTS)HUe%a4vG_EJZ!!-Q%# z5@009t?wTbb=Xq6Zf439JnP#FnH=bgkZ;&$q+r(&E^6d2CKFrDuk4By8)#~|v3sQv z%s0s6`Lg@vH7@-uV^*a)LcW}WRlsv}+D(q^iVdixT?j}qA~4zTJTQ~`=}_Br-yMGk z#VopES(+b%I!DPm<1mYi=Eu+vBRYqZdf!=4#{D^#dA8&u<_Q2|eM=bwJ*@=gVjXGt z7t4HAUfHm5)=O^Txfbz#O-Uqe?S0|$5;6<96~KGi6v_v4i$|6FI(uY z2L|UY=0A8aP)eDacx%=J383IIwScp2=+cLMi@oa1!<&O&6%r-X3(mHc3WmJwXxLQs zjej2#zWuURZdqb__y>A;PuF+?Ph0GJhkl;X?)cO9ktBOYz=WNN+WNQg_^j0x1N^{6 z?1y9CpAueUU1QS;&>c6ddoJ43S07uX)NhjxwW( zFExO9q_>jZ!JP>lR8qgkkU{aHJHG$<=bCGCF{~wcbrF!A>1^KglM`w|fe?CJ0}qdY z-HN^N`tk}fI}jEmTl+ZSSRB?joYsIDu=A7tG63F_YHTwU@R3{x2_$23Q3vZ4qK;E1 z@1^373^3u{U^Po_LXnI(7HYn!Ap1+4E#5apY6c}yZ`?{8L8vi`I!W3~OEpkeoFJsb zKZYfrmA~v5y};3ys2>H-XDbX>1m%^Oa5zhEA1%Sd-~&4J%y zVq|(+Ck?t;vg08R)f|G)3(YJqd+36Ms!-g8ax6E?h3^&qxG2bs`@Guk%l*}|$67u2 zh?}(Yy!a*|QJ72#urK~`F}{H-P4<^k@6R{rN9u)L<2}Rel9hwA3X<<;kJ~Y~v;zn` zU1XEZYTL7fhVkJwf*dUd)0BfE&-h0!%~>kOg0IUEY7kxZ(^i|hO2N^3UVN&F;?)`2 zMJCjinS?xjADK1l0(+58$|Tj0j=Z$Q5%eAHD>SGalByzUKt5v|`??^e-;II0g-TtX z>)uv0J7bz&iV~cQEZ&xN!*OI?=?+!(h>j28Kv@f~YIjE7^B|P>Tybg_ksE3SQ`TmF z%XSIJJ+cN5=YtTCYukVlIZfmhMXR}@l1jkm z;6=X@vh!jK; zwri}4qWdfZdc?2QNpZT}4^F&gdF}0)Z)8UlR-E|gPE~7vYqGzSe6DV|GSX1IDwO8D zG|?u{*9UO0yRR3_(Xu&%P)d!m(y5gFyU3KXllgq%>&g8rstil_)$_y)gbL|WVXe2- zdKpVHXGqsPkf9Wb>!UV%#w0+r2|dw8A8u%{8&@Vt`2IW^s$O!87o}{t|Mwn_OBeKK z2&~!FGYw%`?EVr{Sw}eewp>&vVZ=6RgpaBf${|83cRG0It$^pOR6wn_%E*hXR8>Ur z)ith)v}}LV?X17ZG9p6l`u;GV^BJ|2uTZu^Gr2*hf$bS+zds%@69A;;s(g0Z?Zevv zl~rmn)3HvWqSJIS-Cb%xdP80Y<(q#~^OUs-<&Aq6F3rWJNXkDy{d(y~lPJ$2(>%n8 z+U=cuWdVy^|1?{ZfQj}0jB;A8eQ=*Jl5yXI-Lp)-g7N!qPV@|Q%FN>kELU%%1 z473RuZ4&I5oKJaf$esU1^b`cW>jaMZXWsLPLUbIWx}=LEr*-VV7sN@yTQ& zQ0Ahb=l|7zDGQ<9VCXLCe*pXY>kCuW zP8NA{VC>@G9sY?PWoIm*-Cn`%?qBs_|7n(Cx=7Ft>}e%r|5yK9gmy&K(&K+8^^Yf+ zD+Er}k};$Czxp>Iw3o3!UH&PE{PTwaHwpZh{idD!k6!;JwoE0V-BKxG{)`$6nN5tG z!rfnLIG>%=9bdbZWmZO?2?)C)DoR5V_V1rgpCS{9E?#d<=6(Iv5PYy$>N4M6kdno3 z(VdC$%H3ZEN<$`AGjBK;L``>0h>ULXAGj4aY8}Nlq^5%TOCsv}pB`D6PIz_f`PRET z67o_T{qhBkN;|tlk3ZloW4rnjQ|Hay5{*OXx^)&Ox~bV|r0l&Bk4Wv~S;;Gp={zFT z+&6dKPLw^tEC?yYc4jF^82b?$QIzTy9qwOV=8@zcDs%~CKT*kbsU=wagK{X}h?GEs zG5^u>dgb=gt##lkk}Vmu`aZdjFK5WOv_q|9ziV%{!XZyu$wRs-9~M-{Q+$HLS^+>4 z%Y#nFEC}uqm)3E@o}2n2xNgxl&h0p0tIjEX-Uk!6(XV;%s9pSdz5j*i8n`b;s~8|@ zGg(`4l}_*louD=4FaKRyCL&pyPCQc)i8D|2%-gh7()OTJ5>oO8I0ymbxa6mE)e6() zD+_hILC(=%^eP^jWLW+$LtIcw)67!bnGIsR(MdjP^IA~!VI)=-xdctLk|Y)(EhheO z7Rj1iwijIxL%edD{>uFfz&nbXV43 zyy;seG&NN#%Gg;@`S!oE7GQqyVNPVh?FnW|E&{ZM)WY+n@~67xg+4)Uhs-zrN2K4x zog+(hF=C2j2&ku+emq(!l6CrbK!25R{ z=P3@Flmj=e6dF{;$_A1s1d@8Xhh#%Wes$W7|O>I|hi+vzem{peaHdOAGeyZ}X zN0}Y<9qm4K3$1df2PGeAO1|RX<>)He03C(G4<6O_hrqoW+AW;oR_?Zmzv734co;&Y zfqfnYqFFm`kdA#Zd2smhubX@RaYC|4$IEZ4+gO9eKQTGV8nY=goDnWS@X{A{t>=5v z<4Gr|;}cY>I3xtK`}Bwf2hz~c@VbjOtM*4Tq!M6&xb+VnPXj=rM{PKxQfaK4a$Udf z31rs|yNOCYn0#Kf3sXasTC^;8peGKJ>lpD?eVZF#WC!rLN|J$)a60nAB!7swWW@BQ z>Eyx{{@@v+lwIZzG`>@1A5E2Fd;`m$t6Ldb z4|)Y15Z-Q&?cIzy2IL=pvMWx4rle3r5apNENYF-vN6UA8aIt{+xTA*Ts#)W2sk6g9z#P zX|}RLos+JO)Q9KjID>6SvB>$YD-zl3*e+!181*UH(wr||nxjV7SS_7tO1=xi_EQ`x zMX7h;IhqIeQn{TwW%K|;DqV<8z99W=>0pDJC!c%il_^Ew zeO&Wr-orm;!BvC6x7;a0?Z0zR8UY@OpDZE*wB{Hx6yKNfTR8`Xh6ncjcO-R!kuW6v zt-^JQ>_n)Q%I?gOy5zrl#3?moR39=y^U)?odO8TeOk{Kd&8_jA)dWNtcjc&s?U47(j zNYWFGqJ_n%(HE^wtX;stKO2$H0G1_q0C(P%HQhdgg^1;cZ40%H_q>%$hjxah=)r1H zAhD1m$$#vBWGZ2#-Q9YA&a3`}Ea3`^}2cH=+kAM$wFh^oYRhxt(tTtvbI>4b+iu>AG z%D}-eAKZE>!dKgcjZ`47F z$SDzJQm!Zqoku#dMy`)=#PCEZu#y4c`nPu7_m#=cr1qZ=rWb_40^NmgpScV&X%_(a zIkg&_Rzt)OZ#yqTacy?Ch@`0};gfZqY368l5SUB;Z+(NVk=n zPWN2Q9hD^q7SYUuYo%rtJ2orv-2eG2;J${orBP> z^gfwTSM>je@R;CB;oMBmc=rDRLz?z!Y;Z-K^iP%WFO?1n&NODZ_0=C({X3LQ?x`5u z^1Sxx-*oSvmQ2MsfLC^Dj0TAje}-8J6-69}KtCICr*Hqm!E^!i!YWiC5Yp3@jj*!tfg zF(kw$_{INjl`r-4pZESH&u_F&W&WG-`Ci7;UYHbR%ABybA0q_F;D~C6c{+Yv>FCOT z8935~F!6-*a?YJ$UBkN7@hQx8#zAZTnr|FEM|Y@yMNdU8AK*;5|M>{Gdm2LrvW4CI z4d1I7AXTeFG*hs_^aYn3LcqE4F%9_7uYaQyNnl{^3%_4JI@Q@X{zGS3Lb!7$v#eS7 z3>Tt%TN68BL;L%`7H`rG=RJfSD~P_AN#Z35_+wiCe~S7UYw!fOEG1Bfsx?`WQpDSE z|8EZd3!Xo`qsUDn$Z>=0_%t6ES`e6Tq|jJh2K&-^Se+eMt-cT>uNnpQZ_7N_)o8oP8o!VKj=4ua0M90 zIz$F;h8nAWd!4MfgHR#Zj6#7P>pw&?0C`IK#N{xJ-=tSJ3AjAx`GT_5!TfEt79n2Q zdNd⪻xrO#OiT&ryeC?AVN&X9e3f+AIL-!gpXv^o|pRcCjl~-9_iKs#E?RgXk9v# zu3k2BjFocUg7QD}b-+akFCqOrWoM~{7qTYDQJA4^^etgj zqE4ZIa69n~_ytIl1XZi#16j(dkC*4bgy}lrA+h|^z+8|a@bo!VBDu4EE}e#yCdm}f z|FnrWYYDHPK#0CdpULQ#CM0CE=Ud=>oR+nIRjD6;r0W$!ssDfC}DvT=e%`T*_yGbfS#;ED3^G1cq7&ye7H18Ajijj4nY#z5!!9| zlJr?3Sxjdkbek&M&by5MkEQSEc(t@H_M1ljaL=K5`ZfxhYwFsE8$;8+JA781eBCjq z(8c!j0b6t5y-KSI&3RBgmJ0-cKUvwu#d8{!f8jc7;CT$6`+&Rc-7H>q=>oIQhjy z!xV;2fOkw7z%jqs-d&HxPK|KFYp;A5puNu38pN`QpNo}#XzM2j73`OPnA;rAg~Os- zajJj>vXzg!BUqv{YaKp!FwfgvlXR$mi1ldI#J%iu2{(KVRmgjDj|et)^2iq134(h^ zx|dQMOzO0vTHD!Hde*`|87fOKl#qO{9~?}xkRKN!Iql-pid4JWH+kIfNqoh<@y=W( zu`kc~e=vB*9uZ}OdD6g+O+A;7S5rftxrwHCE|&r{8qNWhsU{X%AX!tkB5q5Rqeh0M z!9eh#w}XYvJ!-RRjqW{CT!eF(&J_eggiZ-+PkCYRU+HokSE~{#qw)UoE>bNLn9qL; zH@R}YX%9q9gHzM4_k0{=N{b827Jggt1XSFF&k-ncvd~Je2;icw_ zu6HKl&sF%G&lTs1zpr3*@Fl{Cd))!B5<)LpLui?Db~as-1_E8cmFN{yLI1oru;=Mzf+2s zNx}3RV8#vfSc@{QAI{EFj70`U(HFn9$ru!bRx1j^JwS$4=61BdMl*%9MvLi_iBaOS zViSz}^fSLsG8)}GkQ!o1c`&&=%(v)HEwm3TrBaaxw9&8yO}t}DkVW(F|?nhP&lmu(sf^h!#Ijf1=EH^MyJUkhWrbcm4=_p2TcY+27WRC9CH6>Ntn9X zJ7*?XAyUczgn@4pSt>b>G1ZNm#@aO+Zt4noLprE#X21g3R)R6u9n;8`saa#OjYv3f?~( zmDQ=xo!b!16Q2|tAZKmnRv9SL#=Ma)AtDb-`8R8Vg%90AH&48SfPa*C4e?dy zh1f@gzLIOihf&tOrICL)XGjhq#|KM56l3cE$LIy!l-5KX6C@BhA5q-mT;XJ2)UVL` zCn=?#d_U>r2)nj*(vgEbT`+QoRUMyhALm*4B4F?w0H4*7(b9;|mtzc#moLi!LEQZQ zZ4266!r%o~5C^Y`qa6DP8^PRQ`y~G386OYT65_kq-8Yyiq-XItA-}6q#vhdFV1o-3 zMTNklrA@7Mh=(broa3xfwz=IxtTd(BrJ#xWW|pp`Pio7_*S5z36q1EgvU=sj^Jwa4 z#i^T>Y$ z2QI)ook7?&#OWc|MoeJ~*Q;h$ku%RK>g57VA}Xipxs}_uHDEhdcv=it8TI)k9Zy&g z)FNWRxM5cH{f}aV(xw@357=@c9Kh*JSCw+aI^gsU9D(J$5`EFELcE*z%gjMLmYYQ? zPbb6ZDab4}48l&MeD4OmZn@yKF70+}wbOJcX*kAj9gSkFeo&HQ$L_$;EmRGko)Pkr z;t;aBMCQy9#@Mv>h1e%SSLR3~y^&#NLm8@huLEX~&i}i8=-9D5*l^(1TsW2<3x$^j zl!ccbI91eZquv_CRJb!qlF`C?X%+u9I3XzOut`E&`4|=^`XBjc?rz}2PBfx=WTYNDu z$`=V4DuNG;xd+bgSbFJI65QHImm(u!gg^rO]) / ` can be used for the rate. TODO: link - - -*Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues with sparse deltas, like making it harder to guess the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).* - -`rate()` and `increase()` will be extended to support delta metrics too. If the `__temporality__` is `delta`, execute delta-specific logic instead of the current cumulative logic. For consistency, the delta-specific logic will keep the intention of the rate/increase functions - that is, estimate the rate/increase over the selected range given the samples in the range, extrapolating if the samples do not align with the start and end of the range. - -`irate()` will also be extended to support delta metrics. +For this initial proposal, existing functions will be used for querying deltas. -Having functions transparently handle the temporality simplifies the user experience - users do not need to know the temporality of a series for querying, and means queries don't need to be rewriten when migrating between cumulative and delta metrics. +`rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[]) / ` can be used for the rate. `metric / interval` can also be used to calculate a rate if the ingestion interval is known. -`resets()` does not apply to delta metrics, however, so will return no results plus a warning in this case. +Having different functions for delta and cumulative counters mean that if the temporality of a metric changes, queries will have to be updated. -#### rate() calculation +Possible improvements to rate/increase calculations and user experience can be found in the Function overloading section below. -In general: `sum of second to last sample values / (last sample ts - first sample ts)) * range`. We skip the value of the first sample as we do not know its interval. +#### Querying range misalignment -The current `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. +One caveat with using `sum_over_time` is that the actual range covered by the sum could be different from the query range. For the ranges to match, the query range needs to be a multiple of the collection interval, which Prometheus does not enforce. Also, this finds the rate between the start time of the first sample and the end time of the last sample, which won't always match the start and end times of the query. -With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. +Below are a couple of examples. -We could just not try and predict the start/end of the series and assume the series continues to extend to beyond the samples in the range. However, not predicting the start and end of the series could inflate the rate/increase value, which can be especially problematic during rollouts when old series are replaced by new ones. +**Example 1** -Assuming `rate()` only has information about the sample within the range, guessing the start and end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has started ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series). +* S1: StartTimeUnixNano: T0, TimeUnixNano: T2, Value: 5 +* S2: StartTimeUnixNano: T2, TimeUnixNano: T4, Value: 1 +* S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 -To calculate the interval, use the average spacing between samples. +And `sum_over_time()` was executed between T1 and T5. -Downsides: +As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. -* This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). -* Harder to predict the start and end of the series vs cumulative. -* The average spacing may not be a good estimation for the ingestion interval, since delta metrics can be sparse. +**Example 2** -#### Non-approximation +* S1: StartTimeUnixNano: T0, TimeUnixNano: T5, Value: 10 -While `rate()` and `increase()` will be consistent with cumulative case, there are downsides as mentioned above. Therefore there may be cases where approximating the rate/increase over the selected range is unwanted for delta metrics. +`sum_over_time()` between T0 and T5 will get 10. Divided by 5 for the rate results in 2. -Users may prefer "non-approximating" behaviour that just gives them the sum of the sample values within the range. This can be accomplished with `sum_over_time()`, though this does not always accurately give the increase within the requested range. +However, if you only query between T4 and T5, the rate would be 10/1 = 1 , and queries between earlier times (T0-T1, T1-T2 etc.) will have a rate of zero. These results may be misleading. -`sum_over_time()` does not work for cumulative metrics, so a warning should be returned in this case. One downside is that this could make migrating from delta to cumulative metrics harder, since `sum_over_time()` queries would need to be rewritten, and users wanting to use `sum_over_time()` will need to know the temporality of their metrics. +#### Function warnings -One possible solution would to have a function that does `sum_over_time()` for deltas and the cumulative equivalent too (this requires subtracting the latest sample before the start of the range with the last sample in the range). This is outside the scope of this design, however. +To help users use the correct functions, warnings will be added if the metric type/temporality does not match the types that should be used with the function. -TODO: mixed samples +* `rate()` and `increase()` will warn if `__type__="gauge"` or `__temporality__="delta"` +* `sum_over_time()` will warn if `__temporality__="cumulative"` ## Possible future extensions @@ -260,6 +257,8 @@ CT-per-sample is not a blocker for deltas - before this is ready, `StartTimeUnix Having CT-per-sample can improve the `rate()` calculation - the ingestion interval for each sample will be directly available, rather than having to guess the interval based on gaps. It also means a single sample in the range can result in a result from `rate()` as the range will effectively have an additional point at `StartTimeUnixNano`. +There are unknowns over the performance and storage of essentially doubling the number of samples with this approach. + ### Inject zeroes for StartTimeUnixNano [CreatedAt timestamps can be injected as 0-valued samples](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection). Similar could be done for StartTimeUnixNano. @@ -267,111 +266,135 @@ Having CT-per-sample can improve the `rate()` calculation - the ingestion interv CT-per-sample is a better solution overall as it links the start timestamp with the sample. It makes it easier to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. If CT-per-sample takes too long, this could be a temporary solution. -TODO: also mention space and performance of additional sample -### Lookahead and lookbehind of range -TODO: move to querying section +It's possible for the StartTimeUnixNano of a sample to be the same as the TimeUnixNano of the preceding sample; care would need to be taken to not overwrite the non-zero sample value. -The reason why `increase()`/`rate()` need extrapolation to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. +### Rate calculation extensions -To work out the increase more accurately, they would also have to look at the sample before and the sample after the range to see if there are samples that partially overlap with the range - in that case the partial overlaps should be added to the increase. +Querying deltas outlined the caveats of using `sum_over_time(...[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. Additionally, this section references the [Extended range selectors semantics proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md) which introduces extensions to range selectors, in particular for `rate()` and `increase()` for cumulative counters. -This could be a new function, or changing the `rate()` function (it could be dangerous to adjust `rate()`/`increase()` though as they’re so widely used that users may be dependent on their current behaviour even if they are “less accurate”). +#### Similar logic to cumulative metrics -## Discarded alternatives +For cumulative counters, `increase()` works by subtracting the first sample from the last sample in the range, adjusting for counter resets, and then extrapolating to estimate the increase for the entire range. The extrapolation is required as the first and last samples are unlikely to perfectly align with the start and end of the range, and therefore just taking the difference between the two is likely to be an underestimation of the increase for the range. rate() divides the result of increase() by the range. This gives an estimate of the increase or rate of the selected range. -### Ingesting deltas alternatives +For consistency, we could emulate that for deltas. In general the calculation would be: `sum of second to last sample values / (last sample ts - first sample ts))` for delta rate, and multiply by `range` for delta increase. We skip the value of the first sample as we do not know its interval. A slight alternative is to partially include the first sample to avoid discarding information, using the average interval between samples to estimate how much of it is within the selected range. -#### Treat as “mini-cumulative” -Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. +The cumulative `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. We could just not try and predict the start/end of the series and assume the series continues to extend to beyond the samples in the range. However, not predicting the start and end of the series could inflate the rate/increase value, which can be especially problematic during rollouts when old series are replaced by new ones. -This requires CT-per-sample to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. +Assuming the delta rate function only has information about the sample within the range, guessing the start and end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has started or ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series). To calculate the interval, use the average spacing between samples. -Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. rate() assumes samples in a range are equally spaced to figure out how much to extrapolate, which is less likely to be true for delta samples. TODO: depends on delta type +Downsides: -This also does not work for samples missing StartTimeUnixNano. +* This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). + * A possible adjustment is to just take the single value as the increase for the range. This may be more useful on average than returning no value in the case of a single sample. However, the mix of extrapolation and non-extrapolation logic may end up surprising users. If we do decide to generally extrapolate to fill the whole window, but have this special case for a single datapoint, someone might rely on the non-extrapolation behaviour and get surprised when there are two points and it changes. +* Harder to predict the start and end of the series vs cumulative. +* The average spacing may not be a good estimation for the ingestion interval when delta metrics are sparse. -#### Convert to rate on ingest +##### Lookahead and lookbehind of range -Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixName` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. +The reason why `increase()`/`rate()` need extrapolation for cumulative counters is to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. -The difference is that there is no interval information in the metric name (like :rate1m) as there is no guarantee that the interval from sample to sample stays constant. +To work out the increase more accurately, the functions would also have to look at the sample before and the sample after the range to see if there are samples that partially overlap with the range - in that case the partial overlaps should be added to the increase. -To averages rates over more than the original collection interval, a new time-weighted average function is required to accommdate cases like the collection interval changing and having a query range which isn't a multiple of the interval. +The `smoothed` modifer in the extended range selectors proposal does this for cumulative counters - looking at the points before and after the range to more accurately calculate the rate/increase. We could implement something similar with deltas, though we cannot naively use the propossed smoothed behaviour for deltas. -This would also require zero timestamp injection or CT-per-sample for better rate calculations. +The `smoothed` proposal works by injecting points at the edges of the range. For the start boundary, the injected point will have its value worked out by linearly interpolating between the closest point before the range start and the first point inside the range. -Users might want to convert back to original values (e.g. to sum the original values over time). It can be difficult to reconstruct the original value if the start timestamp is far away (as there are probably limits to how far we could look back). Having CT-per-sample would help in this case, as both the StartTimeUnixNano and the TimeUnixNano would be within the sample. However, in this case it is trivial to convert between the rated and unrated count, so there is no additional benefit of storing as the calculated rate. In that case, we should prefer to store the original value as that would cause less confusion to users who look at the stored values. +![cumulative smoothed example](../assets/2025-03-25_otel-delta-temporality-support/cumulative-smoothed.png) -This also does not work for samples missing StartTimeUnixNano. +That value would be nonesensical for deltas, as the values for delta samples are independent. Additionally, for deltas, to work out the increase, we add all the values up in the range (with some adjustments) vs in the cumulative case where you subtract the first point in the range from the last point. So it makes sense the smoothing behaviour would be different. -### Distinguishing between delta and cumulative metrics alternatives +In the delta case, we would need to work out the proportion of the first sample within the range and update its value. We would use the assumption that the start timestamp for the first sample is equal the the timestamp of the previous sample, and then use the formula `inside value * (inside ts - range start ts) / (inside ts - outside ts)` to adjust the first sample (aka the `inside value`). -#### Add delta `__type__` label values +### Function overloading -Instead of a new `__temporality__` label, extend `__type__` from the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files) with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) +`rate()` and `increase()` could be extended to work transparently with both cumulative and delta metrics. The PromQL engine could check the `__temporality__` label and execute the correct logic. -A downside is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)”` or `__type__=~”delta_.*”` would have to be used. (However, this does not seem like a particularly necessary use case to optimise for.) +This would mean users would not need to know the temporality of their metric to write queries. Users often don’t know or may not be able to control the temporality of a metric (e.g. if they instrument the application, but the metric processing pipeline run by another team changes the temporality). Different sources may also mean different series have different temporalities. -Additionally, combining temporality and type means that every time a new type is added to Prometheus/OTEL, two `__type__` values would have to be added. This is unlikely to happen very often, so only a minor con. +Additionally, it allows for greater query portability and reusability. Published generic dashboards and alert rules (e.g. [Prometheus monitoring mixins](https://monitoring.mixins.dev/)) can be reused for metrics of any temporality, reducing oeprational overhead. -#### Metric naming convention +However, there are several considerations and open questions: -Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. +* There are questions on how to best calculate the rate or increase of delta metrics (see section below), and there is currently ongoing work with extending range selectors for cumulative counters, which should be taken into account when considering how to calcualte delta ([proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md)). +* While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. +* The increased internal complexity could end up being more confusing. +* Migration between delta and cumulative temporality may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). One possible change could be to attempt to stitch the cumulative and delta series together and return a single result. +* There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. +* While updating queries could be tedious, it's an explicit and informed process, and just doing it will solve the problem. +* Once we start with overloading functions, users may ask for more of that e.g. should we change `sum_over_time()` to also allow calculating the increase of cumulative metrics rather than just summing samples together. Where would the line be in terms of which functions should be overloaded or not? One option would be to only allow `rate()` and `increase()` to be overloaded, as they are the most popular functions that would be used with counters. -### Querying deltas alternatives +Function overloading could also technically work f OTEL deltas are ingested as Prometheus gauges and the `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on actual gauges (e.g. max cpu), not add any warnings, and produce results that don’t really make sense. -#### Do sum_over_time() / range for delta `rate()` implementation +#### `rate()` behaviour for deltas -Instead of trying to approximate the rate over the interval, just sum all the samples in the range and divide by the range for `rate()`. This avoids any approximation which may be less effective for delta metrics where the start and end of series is harder to predict, so may be preferred by users. +If we were to implement function overloading for `rate()` and `increase()`, how exactly will it behave for deltas? In this document, a few possible ways to do rate calculation are outlined, each with their own pros and cons. -For cumulative metrics, just taking the samples in the range and not approximating to cover the whole range is a bad approach. In the cumulative case, this would mean just taking the difference between the first and last samples and dividing by the range. As the first and last samples are unlikely to perfectly align with the start and end of the range, taking the difference between the two is likely to be an underestimation of the increase for the range. +Doing extrapolation in a similar way to the current cumulative rate calculation would be the most consistent option, and if users have a mix of delta and cumulative metrics, or migrate from one to another, there are fewer surprises. However, the extrapolating behaviour works less well with deltas, and there are also issues with it on the cumulative side. -For delta metrics, this is less likely to be an underestimation. Non-approximation would mean something different than in the cumulative case - summing all the samples together. It's less likely to be an underestimation because the start time of the first sample could be before the start of the query range. So the total range of the selected samples could be similar to the query range. +Our suggestion for the inital delta implementation is for users to directly use `sum_over_time()` to calculate the increase in a delta metric. We could instead do the `sum_over_time()` calculation for `rate()`/`increase()` if those functions are called on deltas. This could cause confusion if the users are switching between delta and cumulative metrics in some way, and there could be a range misalignment. -Below is an image to demonstrate - the filled in blue squares are the samples within the range, with the lines between the interval for which the sample data was collected. The unfilled blue squad is the start time for the first sample in the range, which is before the start time of the query range, and the total range of the samples is similar to the query range, just offset. +Also to take into account are the new `smoothed` and `anchored` modifiers in the extended range selectors proposal. One possible solution is to +allowing different behaviours depending on the modifiers. -![Range covered by samples vs query range](../assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png) +The closest match in terms of consistent behaviour between deltas and cumulative given the same input to the instrumented counters in the application would be: +* no modifier - Logic as described in Similar logic to cumulative metrics. +* `smoothed` - Logic as described in Lookahead and lookbehind. +* `anchored` - In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas. To get the same result, for deltas we would just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). No samples outside the range need to be considered for deltas. -The actual range covered by the sum could still be different from the query range in the delta case. For the ranges to match, the query range needs to be a multiple of the collection interval, which Prometheus does not enforce. Also, this finds the rate between the start time of the first sample and the end time of the last sample, which won't always match the start and end times of the query. +An adjustment could be to do `sum_over_time()` in the no modifier case too, even though it's less consistent with cumulative behaviour, since the extrapolation behaviour with delta metrics has issues and could end up being more confusing despite being more consistent with the cumulative implementaiton. -Below are a couple of examples. +One problem with reusing the range selector modifiers is that they are more generic than just modifiers for `rate()` and `increase()`, so adding delta-specific logic for these modifiers for `rate()` and `increase()`, may be confusing. -**Example 1** +### `delta_*` functions -* S1: StartTimeUnixNano: T0, TimeUnixNano: T2, Value: 5 -* S2: StartTimeUnixNano: T2, TimeUnixNano: T4, Value: 1 -* S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 +An alternative to function overloading, but allowing more choices on how rate calculation can be done would be to introduce `delta_*` functions like `delta_rate()` and having range selector modifiers. -And `sum_over_time()` was executed between T1 and T5. +This has the problem of having to use different functions for delta and cumulative metrics (so switching cost, possibly poor user experience). -As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. +## Discarded alternatives -**Example 2** +### Ingesting deltas alternatives -* S1: StartTimeUnixNano: T0, TimeUnixNano: T5, Value: 10 +#### Treat as “mini-cumulative” -`sum_over_time()` between T0 and T5 will get 10. Divided by 5 for the rate results in 2. +Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. -However, if you only query between T4 and T5, the rate would be 10/1 = 1 , and queries between earlier times (T0-T1, T1-T2 etc.) will have a rate of zero. These results may be misleading. +This requires CT-per-sample to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. + +Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate, which is less likely to be true for delta samples. TODO: depends on delta type + +This also does not work for samples missing StartTimeUnixNano. + +#### Convert to rate on ingest -For users wanting this behaviour instead of the suggested one (approximating the rate/increase over the selected range), it is still possible do with `sum_over_time()` / ``. +Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixName` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. + +The difference is that there is no interval information in the metric name (like :rate1m) as there is no guarantee that the interval from sample to sample stays constant. + +To averages rates over more than the original collection interval, a new time-weighted average function is required to accommdate cases like the collection interval changing and having a query range which isn't a multiple of the interval. + +This would also require zero timestamp injection or CT-per-sample for better rate calculations. + +Users might want to convert back to original values (e.g. to sum the original values over time). It can be difficult to reconstruct the original value if the start timestamp is far away (as there are probably limits to how far we could look back). Having CT-per-sample would help in this case, as both the StartTimeUnixNano and the TimeUnixNano would be within the sample. However, in this case it is trivial to convert between the rated and unrated count, so there is no additional benefit of storing as the calculated rate. In that case, we should prefer to store the original value as that would cause less confusion to users who look at the stored values. + +This also does not work for samples missing StartTimeUnixNano. -Having `rate()`/`increase()` do different things for cumulative and delta metrics can be confusing (e.g. with deltas and integer samples, you'd always get an integer value if you use `sum_over_time()`, but the same wouldn't be true for cumulative metrics with the current `increase()` behaviour). +### Distinguishing between delta and cumulative metrics alternatives -If we were to add behaviour to do the cumulative version of "sum_over_time", that would likely be in a different function. And then you'd have different functions to do non-approximation for delta and cumulative metrics, which again causes confusion. +#### Add delta `__type__` label values -If we went for this approach first, but then updated `rate()` to lookahead and lookbehind in the long term, users depending on the "non-extrapolation" could be affected. +Instead of a new `__temporality__` label, extend `__type__` from the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files) with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) -One open question is if or how much the "accuracy" problem matters to users. Furthermore, in the case of the second example, where the query range is smaller than the ingestion interval, that may not be a very frequent problem. +A downside is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)”` or `__type__=~”delta_.*”` would have to be used. (However, this does not seem like a particularly necessary use case to optimise for.) -#### Do sum_over_time() / range for delta `rate()` implementation only when StartTimeUnixNano is missing +Additionally, combining temporality and type means that every time a new type is added to Prometheus/OTEL, two `__type__` values would have to be added. This is unlikely to happen very often, so only a minor con. -Use the proposed logic (with interval-based approximation) when StartTimeUnixNano is set, but if it's missing, use sum_over_time() / range. +#### Metric naming convention -With the initial implementation, all delta metrics will essentially have missing "StartTimeUnixNano" since that is discarded on ingestion and not available at query time. +Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. -This also provides a way for users to choose between the approximating and non-approximating query behaviour by setting StartTimeUnixNano or not. However, the users querying the metrics may be different from the operators of the metrics pipeline, and therefore still not have control over the query behaviour. +### Querying deltas alternatives #### Convert to cumulative on query @@ -381,12 +404,6 @@ No function modification needed - all cumulative functions will work for samples However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. -#### Have temporary `delta_rate()` and `delta_increase()` functions - -While the intention is to eventually use `rate()`/`increase()` etc. for both delta and cumulative metrics, initially experimental functions prefixed with `delta_` will be introduced behind a delta-support feature flag. This is to make it clear that these are experimental and the logic could change as we start seeing how they work in real-world scenarios. In the long run, we’d move the logic into `rate()` etc.. - -This may be an unnecessary step, especially if the delta functionality is behind feature flags. - ## Known unknowns ### Native histograms performance From 5dcc9c44ea44b9a2b072bf27ccb31ecbf9ccbad4 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 11 Jul 2025 19:31:35 +0100 Subject: [PATCH 15/34] Clarify Signed-off-by: Fiona Liao --- proposals/2025-03-25_otel-delta-temporality-support.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 2f17e19..71c460d 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -304,7 +304,7 @@ The `smoothed` proposal works by injecting points at the edges of the range. For That value would be nonesensical for deltas, as the values for delta samples are independent. Additionally, for deltas, to work out the increase, we add all the values up in the range (with some adjustments) vs in the cumulative case where you subtract the first point in the range from the last point. So it makes sense the smoothing behaviour would be different. -In the delta case, we would need to work out the proportion of the first sample within the range and update its value. We would use the assumption that the start timestamp for the first sample is equal the the timestamp of the previous sample, and then use the formula `inside value * (inside ts - range start ts) / (inside ts - outside ts)` to adjust the first sample (aka the `inside value`). +In the delta case, the adjustment to samples in the range used for the rate calculation would be to work out the proportion of the first sample within the range and update its value. We would use the assumption that the start timestamp for the first sample is equal the the timestamp of the previous sample, and then use the formula `inside value * (inside ts - range start ts) / (inside ts - outside ts)` to adjust the first sample (aka the `inside value`). ### Function overloading From 7291f03ee0d4663ba67c2ca54423ddb8943f6ff9 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 18 Jul 2025 21:55:21 +0100 Subject: [PATCH 16/34] Update proposal Signed-off-by: Fiona Liao --- ...25-03-25_otel-delta-temporality-support.md | 225 ++++++++++-------- 1 file changed, 126 insertions(+), 99 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 71c460d..d65c062 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -14,12 +14,11 @@ * **Other docs or links:** * [Original design doc](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.0) * [#prometheus-delta-dev](https://cloud-native.slack.com/archives/C08C6CMEUF6) - Slack channel for project - * Additional context - * [OpenTelemetry metrics: A guide to Delta vs. Cumulative temporality trade-offs](https://docs.google.com/document/d/1wpsix2VqEIZlgYDM3FJbfhkyFpMq1jNFrJgXgnoBWsU/edit?tab=t.0#heading=h.wwiu0da6ws68) - * [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q) - * [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) + * [OpenTelemetry metrics: A guide to Delta vs. Cumulative temporality trade-offs](https://docs.google.com/document/d/1wpsix2VqEIZlgYDM3FJbfhkyFpMq1jNFrJgXgnoBWsU/edit?tab=t.0#heading=h.wwiu0da6ws68) + * [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q) + * [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) -This design document proposes adding experimental support for OTEL delta temporality metrics in Prometheus, allowing them be ingested, stored and queried directly. +A proposal for adding experimental support for OTEL delta temporality metrics in Prometheus, allowing them be ingested, stored and queried directly. ## Why @@ -45,6 +44,12 @@ Sparse metrics are more common for delta than cumulative metrics. While delta da Further insights and discussions on delta metrics can be found in [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y), which describes Chronosphere's experience of adding functionality to ingest OTEL delta metrics and query them back with PromQL, and also [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q). +#### Alignment + +The Prometheus scrape collection model deliberately gives you "unaligned" sampling, i.e. targets with the same scrape interval are still scraped at different phases (not all at the full minute, but hashed over the minute). + +The usual case for delta metrics is to collect increments over the collection interval (e.g. 1m), and then send out the collected increments at the full minute. This isn't true in all cases though, for example, the StatsD client libraries emits a delta every time an event happens rather than aggregating, producing unaligned samples (though the StatsD daemon does then aggregate to an aligned interval). + ### Pitfalls of the current solution #### Lack of out of order support @@ -69,8 +74,9 @@ State becomes more complex in distributed cases - if there are multiple OTEL col Cumulative metrics usually need to be wrapped in a `rate()` or `increase()` etc. call to get a useful result. However, it could be confusing that when querying just the metric without any functions, the returned value is not the same as the ingested value. -#### Does not handle sparse metrics well -As mentioned in Background, sparse metrics are more common with delta. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. +#### Poor sparse metrics handling + +Sparse metrics are more common with deltas. This can interact awkwardly with `rate()` - the `rate()` function in Prometheus does not work with only a single datapoint in the range, and assumes even spacing between samples. ## Goals @@ -88,7 +94,7 @@ This document is for Prometheus server maintainers, PromQL maintainers, and Prom ## Non-Goals * Support for ingesting delta metrics via other, non-OTLP means (e.g. replacing push gateway). -* Advanced querying support for deltas (e.g. function overloading for rate()). Given that delta support is new and advanced querying also depends on other experimental features, the best approach - or whether we should even extend querying in any way - is currently unclear. However, this document does explore some possible options. +* Advanced querying support for deltas (e.g. function overloading for `rate()`). Given that delta support is new and advanced querying also depends on other experimental features, the best approach - or whether we should even extend querying - is currently uncertain. However, this document does explore some possible options. These may come in later iterations of delta support. @@ -106,7 +112,7 @@ For the initial implementation, reuse existing chunk encodings. Delta counters will use the standard XOR chunks for float samples. -Delta histograms will use native histogram chunks with the `GaugeType` counter reset hint/header. Currently the counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. A (bucket or total) count drop is detected as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Additionally counter histogram chunks have the invariant that no count ever goes down baked into their implementation. `GaugeType` allows counts to go up and down, and does not cut new chunks on counter resets. +Delta histograms will use native histogram chunks with the `GaugeType` counter reset hint/header. The counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. A (bucket or total) count drop is detected as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Additionally, counter histogram chunks have the invariant that no count ever goes down baked into their implementation. `GaugeType` allows counts to go up and down, and does not cut new chunks on counter resets. ### Delta metric type @@ -114,13 +120,13 @@ It is useful to be able to distinguish between delta and cumulative metrics. Thi As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. -There are two options we are considering for how to type deltas. We propose to add both of these options as feature flags for ingesting deltas: +We propose to add two options as feature flags for ingesting deltas: 1. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges. 2. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series. -We would like to initially offer two options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta-like use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but it has the potential to offer a better user experience in the long run as it allows more precise differenciation between metric types. +We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta-like use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but it has the potential to offer a better user experience in the long run as it allows more precise differentiation. Below we explore the pros and cons of each option in more detail. @@ -136,13 +142,13 @@ When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogra * Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. **Cons** -* One problem with treating deltas as gauges is that gauge means different things in Prometheus and OTEL - in Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge, rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. +* Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. * Gauges are usually aggregated in time by averaging or taking the last value, while deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics (e.g. the [Grafana Explore Metrics](https://github.com/grafana/grafana/blob/main/docs/sources/explore/_index.md) feature). * The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. #### Introduce `__temporality__` label -This option extends the metadata labels proposal which introduces the `__type__` and `__unit__` labels (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. +This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. `--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on the that feature. @@ -151,22 +157,23 @@ When ingesting a delta metric via the OTLP endpoint, the metric type is set to ` Cumulative metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. **Pros** -* Clear distinction between delta metrics and gauge metrics. As discussed in the cons for treating deltas as gauges, it could be confusing treating gauge and delta as the same type. -* Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by using the `__type__` and `__temporality__` labels in Prometheus. -* When instrumenting with the OTEL SDK, you have to explicitly define the type but not temporality. Furthermore, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it can be difficult for users to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See the Querying Deltas section for more discussion.) +* Clear distinction between delta metrics and gauge metrics. +* Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having separate `__type__` and `__temporality__` labels in Prometheus. +* When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See Function overloading for more discussion.) **Cons** * Dependent the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usages for refinement. * Introduces additional complexity to the Prometheus data model. -* Systems or scripts that handle Prometheus metrics may be unware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative. This could result in calculation errors that are hard to notice. +* Systems or scripts that handle Prometheus metrics may be unware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. +* In this initial proposal, metrics with `__temporality__="delta"` will essentially be queried in the same way as Prometheus gauges - both gauges and deltas can be aggregated with `sum_over_time()`, so it may be confusing to have two different "types" with the same querying patterns. ### Metric names -Currently, OTEL metric names are normalised when translated to Prometheus by default ([code](https://github.com/prometheus/otlptranslator/blob/94f535e0c5880f8902ab8c7f13e572cfdcf2f18e/metric_namer.go#L157)). As part of this normalisation, suffixes can be added in some cases. For example, OTEL metrics converted into Prometheus counters (i.e. monotonic cumulative sums in OTEL) have the `__total` suffix added to the metric name, while gauges do not. +OTEL metric names are normalised when translated to Prometheus by default ([code](https://github.com/prometheus/otlptranslator/blob/94f535e0c5880f8902ab8c7f13e572cfdcf2f18e/metric_namer.go#L157)). This includes adding suffixes in some cases. For example, OTEL metrics converted into Prometheus counters (i.e. monotonic cumulative sums in OTEL) have the `__total` suffix added to the metric name, while gauges do not. The `_total` suffix will not be added to OTEL deltas, ingested as either counters with temporality label or gauges. The `_total` suffix is used to help users figure out whether a metric is a counter. As deltas depend on type and unit metadata labels being added, especially in the `--enable-feature=otlp-native-delta-ingestion` case, the `__type__` label will be able to provide the distinction and the suffix is unnecessary. -One downside is that switching between cumulative and delta means metric names will change, affecting dashboards and alerts. However, the current proposal requires different functions for querying delta and cumulative counters anyway. +This means switching between cumulative and delta temporality can result in metric names changing, affecting dashboards and alerts. However, the current proposal requires different functions for querying delta and cumulative counters anyway. ### Monoticity @@ -174,21 +181,17 @@ OTEL sums have a [monoticity property](https://opentelemetry.io/docs/specs/otel/ It is not necessary to detect counter resets for delta metrics - to get the increase over an interval, you can just sum the values over that interval. Therefore, for the `--enable-feature=otlp-native-delta-ingestion` option, where OTEL deltas are converted into Prometheus counters (with `__temporality__` label), non-monotonic delta sums will also be converted in the same way (with `__type__="counter"` and `__temporality__="delta"`). -Downsides include not being to convert delta counters in Prometheus into their cumulative counterparts (e.g. for any possible future querying extensions for deltas). Also, as the monoticity information is lost, if the metrics are later exported back into the OTEL format, all deltas will have to be assumed to be non-monotonic. +This ensures StatsD counters can be ingested as Prometheus counters. [The StatsD receiver sets counters as non monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). Note there has been some debate on whether this should be the case or not ([issue 1](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), [issue 2](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14956)). -However, the alternative of mapping non-monotonic delta counters to gauges would be problematic, as it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Different functions would be needed for non-monotonic counters of differerent temporalities. - -Another alternative is to continue to reject non-monotonic delta counters, but this could prevent the ingestion of StatsD counters. [The StatsD receiver sets counters as non monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md), but there has been some debate on whether this should be the case or not ([issue 1](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), [issue 2](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14956)). - -A possible future enhancement is to add an `__monotonicity__` label along with `__temporality__` for counters. Additionally, if there were a reliable way to have [CreatedAt timestamp](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md) for all cumulative counters, we could consider supporting non-monotonic cumulative counters as well, as at that point the CreatedAt timestamp could be used for working out counter resets instead of decreases in counter value. This may not be feasible or wanted in all cases though. +Consequences include not being to convert delta counters in Prometheus into their cumulative counterparts (e.g. for any possible future querying extensions for deltas). Also, as monoticity information is lost, if the metrics are later exported back into the OTEL format, all deltas will have to be assumed to be non-monotonic. ### Scraping No scraped metrics should have delta temporality as there is no additional benefit over cumulative in this case. To produce delta samples from scrapes, the application being scraped has to keep track of when a scrape is done and resetting the counter. If the scraped value fails to be written to storage, the application will not know about it and therefore cannot correctly calculate the delta for the next scrape. -For federation, if the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has the conversion issues detailed above. +### Federation -We will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the `__temporality__="delta"` label is set. If deltas are converted to gauges, there would not be a way to distinguish deltas from regular gauges so we cannot provide a scrape config. +Federating delta series directly could be usable if there is a constant and known collection interval for the delta series, and the metrics are scraped at least as often as the collection interval. This is not the case for all deltas and the scrape interval cannot be enforced. Therefore we will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the `__temporality__="delta"` label is set. If deltas are converted to gauges, there would not be a way to distinguish deltas from regular gauges so we cannot provide a scrape config. ### Remote write ingestion @@ -204,7 +207,7 @@ Once deltas are ingested into Prometheus, they can be converted back into OTEL m The prometheusreceiver has the same issue described in Scraping regarding possibly misaligned scrape vs delta ingestion intervals. -If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we get more certain that the `__temporality__` label is the right way to go, we should update the receiver to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. +If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we gain more confience that the `__temporality__` label is the correct approach, the receiver should be updated to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. ### Querying deltas @@ -214,13 +217,13 @@ For this initial proposal, existing functions will be used for querying deltas. Having different functions for delta and cumulative counters mean that if the temporality of a metric changes, queries will have to be updated. -Possible improvements to rate/increase calculations and user experience can be found in the Function overloading section below. +Possible improvements to rate/increase calculations and user experience can be found in Rate calculation extensions and Function overloading. -#### Querying range misalignment +Note: With [left-open range selectors](https://prometheus.io/docs/prometheus/3.5/migration/#range-selectors-and-lookback-exclude-samples-coinciding-with-the-left-boundary) introduced in Prometheus 3.0, queries such as `sum_over_time(metric[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. Additionally, this section references the [Extended range selectors semantics proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md) which introduces extensions to range selectors, in particular for `rate()` and `increase()` for cumulative counters. - -#### Similar logic to cumulative metrics +Querying deltas outlined the caveats of using `sum_over_time(...[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. -For cumulative counters, `increase()` works by subtracting the first sample from the last sample in the range, adjusting for counter resets, and then extrapolating to estimate the increase for the entire range. The extrapolation is required as the first and last samples are unlikely to perfectly align with the start and end of the range, and therefore just taking the difference between the two is likely to be an underestimation of the increase for the range. rate() divides the result of increase() by the range. This gives an estimate of the increase or rate of the selected range. - -For consistency, we could emulate that for deltas. In general the calculation would be: `sum of second to last sample values / (last sample ts - first sample ts))` for delta rate, and multiply by `range` for delta increase. We skip the value of the first sample as we do not know its interval. A slight alternative is to partially include the first sample to avoid discarding information, using the average interval between samples to estimate how much of it is within the selected range. - -The cumulative `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. We could just not try and predict the start/end of the series and assume the series continues to extend to beyond the samples in the range. However, not predicting the start and end of the series could inflate the rate/increase value, which can be especially problematic during rollouts when old series are replaced by new ones. - -Assuming the delta rate function only has information about the sample within the range, guessing the start and end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has started or ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series). To calculate the interval, use the average spacing between samples. - -Downsides: - -* This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). - * A possible adjustment is to just take the single value as the increase for the range. This may be more useful on average than returning no value in the case of a single sample. However, the mix of extrapolation and non-extrapolation logic may end up surprising users. If we do decide to generally extrapolate to fill the whole window, but have this special case for a single datapoint, someone might rely on the non-extrapolation behaviour and get surprised when there are two points and it changes. -* Harder to predict the start and end of the series vs cumulative. -* The average spacing may not be a good estimation for the ingestion interval when delta metrics are sparse. +This section assumes knowledge of [Extended range selectors semantics proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md) which introduces the `smoothed` and `anchored` modifers to range selectors, in particular for `rate()` and `increase()` for cumulative counters. ##### Lookahead and lookbehind of range @@ -306,51 +307,88 @@ That value would be nonesensical for deltas, as the values for delta samples are In the delta case, the adjustment to samples in the range used for the rate calculation would be to work out the proportion of the first sample within the range and update its value. We would use the assumption that the start timestamp for the first sample is equal the the timestamp of the previous sample, and then use the formula `inside value * (inside ts - range start ts) / (inside ts - outside ts)` to adjust the first sample (aka the `inside value`). +#### Similar logic to cumulative case + +For cumulative counters, `increase()` works by subtracting the first sample from the last sample in the range, adjusting for counter resets, and then extrapolating to estimate the increase for the entire range. The extrapolation is required as the first and last samples are unlikely to perfectly align with the start and end of the range, and therefore just taking the difference between the two is likely to be an underestimation of the increase for the range. `rate()` divides the result of `increase()` by the range. This gives an estimate of the increase or rate of the selected range. + +For consistency, we could emulate that for deltas. + +First sum all sample values in the range, with the first sample’s value only partially included if it's not completely within the query range. To estimate the proporation of the first sample within the range, assume its interval is the average interval betweens all samples within the range. If the last sample does not align with the end of the time range, the sum should be extrapolated to cover the range until the end boundary. + +The cumulative `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. + +For the delta end boundary extrapolation, we could just not try and predict the end of the series and assume the series continues to extend to beyond the samples in the range. However, not predicting the end of the series could inflate the rate/increase value, which can be especially problematic during rollouts when old series are replaced by new ones. + +Assuming the delta rate function only has information about the sample within the range, guessing the end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series). To calculate the interval, use the average spacing between samples. + +The final result will be the increase over the query range. To calculate the rate, divide the increase by the query range. + +Downsides: + +* This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). + * A possible adjustment is to just take the single value as the increase for the range. This may be more useful on average than returning no value in the case of a single sample. However, the mix of extrapolation and non-extrapolation logic may end up surprising users. If we do decide to generally extrapolate to fill the whole window, but have this special case for a single datapoint, someone might rely on the non-extrapolation behaviour and get surprised when there are two points and it changes. +* Harder to predict the start and end of the series vs cumulative. +* The average spacing may not be a good estimation for the ingestion interval when delta metrics are sparse or or deliberately irregularly spaced (e.g. in the classic statsd use case). +* Additional downsides can be found in [this review comment](https://github.com/prometheus/proposals/pull/48#discussion_r2047990524)] for the proposal. + +Due to the numerous downsides, and the fact that more accurate lookahead/lookbehind techniques are already being explored for cumulative metrics (which means we could likely do something similar for deltas), it is unlikely that this option will actually be implemented. + ### Function overloading `rate()` and `increase()` could be extended to work transparently with both cumulative and delta metrics. The PromQL engine could check the `__temporality__` label and execute the correct logic. -This would mean users would not need to know the temporality of their metric to write queries. Users often don’t know or may not be able to control the temporality of a metric (e.g. if they instrument the application, but the metric processing pipeline run by another team changes the temporality). Different sources may also mean different series have different temporalities. +Pros: -Additionally, it allows for greater query portability and reusability. Published generic dashboards and alert rules (e.g. [Prometheus monitoring mixins](https://monitoring.mixins.dev/)) can be reused for metrics of any temporality, reducing oeprational overhead. +* Users would not need to know the temporality of their metric to write queries. Users often don’t know or may not be able to control the temporality of a metric (e.g. if they instrument the application, but the metric processing pipeline run by another team changes the temporality). +* This is helpful when there are different sources which ingest metrics with different temporalities, as a single function can be used for all cases. +* Greater query portability and reusability. Published generic dashboards and alert rules (e.g. [Prometheus monitoring mixins](https://monitoring.mixins.dev/)) can be reused for metrics of any temporality, reducing operational overhead. -However, there are several considerations and open questions: +Cons: -* There are questions on how to best calculate the rate or increase of delta metrics (see section below), and there is currently ongoing work with extending range selectors for cumulative counters, which should be taken into account when considering how to calcualte delta ([proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md)). -* While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. * The increased internal complexity could end up being more confusing. -* Migration between delta and cumulative temporality may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). One possible change could be to attempt to stitch the cumulative and delta series together and return a single result. +* Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). + * One possible extension could be to stitch the cumulative and delta series together and return a single result. * There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. -* While updating queries could be tedious, it's an explicit and informed process, and just doing it will solve the problem. + +Open questions and considerations: + +* While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. +* There are open questions on how to best calculate the rate or increase of delta metrics (see `rate()` behaviour for deltas below), and there is currently ongoing work with [extending range selectors for cumulative counters](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md), which should be taken into account for deltas too. * Once we start with overloading functions, users may ask for more of that e.g. should we change `sum_over_time()` to also allow calculating the increase of cumulative metrics rather than just summing samples together. Where would the line be in terms of which functions should be overloaded or not? One option would be to only allow `rate()` and `increase()` to be overloaded, as they are the most popular functions that would be used with counters. -Function overloading could also technically work f OTEL deltas are ingested as Prometheus gauges and the `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on actual gauges (e.g. max cpu), not add any warnings, and produce results that don’t really make sense. +Function overloading could also technically work if OTEL deltas are ingested as Prometheus gauges and the `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on actual gauges (e.g. max cpu), not add any warnings, and produce nonsensical results. #### `rate()` behaviour for deltas -If we were to implement function overloading for `rate()` and `increase()`, how exactly will it behave for deltas? In this document, a few possible ways to do rate calculation are outlined, each with their own pros and cons. +If we were to implement function overloading for `rate()` and `increase()`, how exactly will it behave for deltas? A few possible ways to do rate calculation have been outlined, each with their own pros and cons. -Doing extrapolation in a similar way to the current cumulative rate calculation would be the most consistent option, and if users have a mix of delta and cumulative metrics, or migrate from one to another, there are fewer surprises. However, the extrapolating behaviour works less well with deltas, and there are also issues with it on the cumulative side. +Also to take into account are the new `smoothed` and `anchored` modifiers in the extended range selectors proposal. -Our suggestion for the inital delta implementation is for users to directly use `sum_over_time()` to calculate the increase in a delta metric. We could instead do the `sum_over_time()` calculation for `rate()`/`increase()` if those functions are called on deltas. This could cause confusion if the users are switching between delta and cumulative metrics in some way, and there could be a range misalignment. +The current proposed solution would be: -Also to take into account are the new `smoothed` and `anchored` modifiers in the extended range selectors proposal. One possible solution is to -allowing different behaviours depending on the modifiers. - -The closest match in terms of consistent behaviour between deltas and cumulative given the same input to the instrumented counters in the application would be: -* no modifier - Logic as described in Similar logic to cumulative metrics. +* no modifier - just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). +* `anchored` - same as no modifer. In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas. To get the same output in the cumulative and delta cases given the same input to the initial instrumented counter, the delta case should use `sum_over_time()`. * `smoothed` - Logic as described in Lookahead and lookbehind. -* `anchored` - In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas. To get the same result, for deltas we would just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). No samples outside the range need to be considered for deltas. -An adjustment could be to do `sum_over_time()` in the no modifier case too, even though it's less consistent with cumulative behaviour, since the extrapolation behaviour with delta metrics has issues and could end up being more confusing despite being more consistent with the cumulative implementaiton. +For the no modifier case, the most consistent behaviour with to cumulative metrics would be do implement what's describe in Similar logic to cumulative case. This could result in fewer surprises if switching between delta and cumulative. However, the extrapolating behaviour does not work well for deltas in many cases, so it's unlikely we will go down that route. + +One problem with reusing the range selector modifiers is that they are more generic than just modifiers for `rate()` and `increase()`, so adding delta-specific logic for these modifiers for `rate()` and `increase()` may be confusing. -One problem with reusing the range selector modifiers is that they are more generic than just modifiers for `rate()` and `increase()`, so adding delta-specific logic for these modifiers for `rate()` and `increase()`, may be confusing. +#### How to proceed + +Before committing to moving forward with function overloading, we should first gain practical experience with the use of `sum_over_time()` for delta metrics and see if there's a real need for overloading, and observe how the `smoothed` and `anchored` modifiers work in practice for cumulative metrics. ### `delta_*` functions An alternative to function overloading, but allowing more choices on how rate calculation can be done would be to introduce `delta_*` functions like `delta_rate()` and having range selector modifiers. -This has the problem of having to use different functions for delta and cumulative metrics (so switching cost, possibly poor user experience). +This has the problem of having to use different functions for delta and cumulative metrics (switching cost, possibly poor user experience). + +### `__monotonicity__` label + +A possible future enhancement is to add an `__monotonicity__` label along with `__temporality__` for counters. + +Additionally, if there were a reliable way to have [Created Timestamp](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md) for all cumulative counters, we could consider supporting non-monotonic cumulative counters as well, as at that point the created timestamp could be used for working out counter resets instead of decreases in counter value. This may not be feasible or wanted in all cases though. ## Discarded alternatives @@ -380,6 +418,10 @@ Users might want to convert back to original values (e.g. to sum the original va This also does not work for samples missing StartTimeUnixNano. +#### Map non-monotonic delta counters to gauges + +Mapping non-monotonic delta counters to gauges would be problematic, as it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Different functions would be needed for non-monotonic counters of differerent temporalities. + ### Distinguishing between delta and cumulative metrics alternatives #### Add delta `__type__` label values @@ -400,7 +442,7 @@ Have a convention for naming metrics e.g. appending `_delta_counter` to a metric Delta to cumulative conversion at query time doesn’t have the same out of order issues as conversion at ingest. When a query is executed, it uses a fixed snapshot of data. The order the data was ingested does not matter, the cumulative values are correctly calculated by processing the samples in timestamp-order. -No function modification needed - all cumulative functions will work for samples ingested as deltas. +No function modification is needed - all cumulative functions will work for samples ingested as deltas. However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. @@ -410,39 +452,24 @@ However, it can be confusing for users that the delta samples they write are tra To work out the delta for all the cumulative native histograms in an range, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the range, all of the buckets in all of the samples need to be summed, which is less efficient at query time. -## Risks -Experimental -what if people use? -more risky, why it's not the default -sum_over_time should always work - - ## Implementation Plan -### Milestone 1: Primitive support for delta ingestion - -Behind a feature flag (`otlp-native-delta-ingestion`), allow OTLP metrics with delta temporality to be ingested and stored as-is, with metric type unknown and no additional labels. To get "increase" or "rate", `sum_over_time(metric[] / )` can be used. - -Having this simple implementation without changing any PromQL functions allow us to get some form of delta ingestion out there gather some feedback to decide the best way to go further. - -PR: https://github.com/prometheus/prometheus/pull/16360 - -### Milestone 2: Introduce temporality label +### 1. Experimental feature flags for OTLP delta ingestion -The second milestone depends on the type and unit metadata proposal being implemented. +Add the `--enable-feature=otlp-native-delta-ingestion` and `otlp-deltas-as-gauge` feature flags as described in Delta metric types to add appropiate types and flags to series on ingestion. -Introduce the `__temporality__` label, similar to how the `__type__` and `__unit__` labels have been added. +Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any types to deltas. -Add `__temporality__="delta"` to delta metrics ingested via the OTLP endpoint (still under the `otlp-native-delta-ingestion` feature flag). +### 2. Function warnings -### Milestone 3: Temporality-aware functions +Add function warnings when a function is used with series of wrong type or temporality as described in Function warnings. -Update functions to use the `__temporality__` label. +There are already warnings if `rate()`/`increase()` are used without the `__type__="counter"` label: https://github.com/prometheus/prometheus/pull/16632. -Update `rate()`, `increase()` and `idelta()` functions to support delta metrics. +### 3. Update documentation -Add warnings to `resets()` if used with delta metrics, and `sum_over_time()` with cumualtive metrics. +Update documentation explaining new experimental delta functionality, including recommended configs for filtering out delta metrics from scraping and remote write. -Temporality-aware functions should be under a `temporality-aware-functions` feature flag. +### 4. Review deltas in practice and experiment with possible future extensions -Also filter out metrics with `__temporality__="delta"` from the federation endpoint. +Review how deltas work in practice using the current approach, and use experience and feedback to decide whether any of the potential extensions are necessary, and whether to discontinue one of the two options for delta ingestion (gauges vs. temporality label). From 83aa03a433e31b8848e53d385de078262408fecd Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 18 Jul 2025 22:14:34 +0100 Subject: [PATCH 17/34] Update contributors Signed-off-by: Fiona Liao --- proposals/2025-03-25_otel-delta-temporality-support.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index d65c062..8122a25 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -3,8 +3,10 @@ * **Owners:** * @fionaliao + +* **Contributors:** * Initial design started by @ArthurSens and @sh0rez - * TODO: add others from delta wg + * Delta WG contributors: @ArthurSens, @enisoc and @subvocal * **Implementation Status:** `Partially implemented` @@ -398,9 +400,9 @@ Additionally, if there were a reliable way to have [Created Timestamp](https://g Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. -This requires CT-per-sample to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. +This requires CT-per-sample (or some kind of precise CT tracking) to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. -Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate, which is less likely to be true for delta samples. TODO: depends on delta type +Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate, which is not always true for delta samples. This also does not work for samples missing StartTimeUnixNano. @@ -422,7 +424,7 @@ This also does not work for samples missing StartTimeUnixNano. Mapping non-monotonic delta counters to gauges would be problematic, as it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Different functions would be needed for non-monotonic counters of differerent temporalities. -### Distinguishing between delta and cumulative metrics alternatives +### Delta metric type alternatives #### Add delta `__type__` label values From 49935637bdf559c236fa80960a4727a7e9407ab8 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 18 Jul 2025 22:16:52 +0100 Subject: [PATCH 18/34] Make sure referenced sections are linked Signed-off-by: Fiona Liao --- ...25-03-25_otel-delta-temporality-support.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index 8122a25..bc1cf6d 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -28,7 +28,7 @@ Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167[). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. -The cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the Pitfalls section below. +The cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the [Pitfalls section](#pitfalls-of-the-current-solution) below. Prometheus' goal of becoming the best OTEL metrics backend means it should improve its support for delta metrics, allowing them to be ingested and stored without being transformed into cumulative. @@ -154,14 +154,14 @@ This option extends the metadata labels proposal (PROM-39). An additional `__tem `--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on the that feature. -When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`), and the `__temporality__="delta"` label will be added. As mentioned in the Chunks section, `GaugeType` should still be the counter reset hint/header. +When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`), and the `__temporality__="delta"` label will be added. As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. Cumulative metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. **Pros** * Clear distinction between delta metrics and gauge metrics. * Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having separate `__type__` and `__temporality__` labels in Prometheus. -* When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See Function overloading for more discussion.) +* When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) **Cons** * Dependent the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usages for refinement. @@ -207,7 +207,7 @@ Prometheus has metric metadata as part of its metric model, which include the ty Once deltas are ingested into Prometheus, they can be converted back into OTEL metrics by the prometheusreceiver (scrape) and [prometheusremotewritereceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusremotewritereceiver) (push). -The prometheusreceiver has the same issue described in Scraping regarding possibly misaligned scrape vs delta ingestion intervals. +The prometheusreceiver has the same issue described in [Scraping](#scraping) regarding possibly misaligned scrape vs delta ingestion intervals. If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we gain more confience that the `__temporality__` label is the correct approach, the receiver should be updated to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. @@ -219,7 +219,7 @@ For this initial proposal, existing functions will be used for querying deltas. Having different functions for delta and cumulative counters mean that if the temporality of a metric changes, queries will have to be updated. -Possible improvements to rate/increase calculations and user experience can be found in Rate calculation extensions and Function overloading. +Possible improvements to rate/increase calculations and user experience can be found in [Rate calculation extensions](#rate-calculation-extensions) and [Function overloading](#function-overloading). Note: With [left-open range selectors](https://prometheus.io/docs/prometheus/3.5/migration/#range-selectors-and-lookback-exclude-samples-coinciding-with-the-left-boundary) introduced in Prometheus 3.0, queries such as `sum_over_time(metric[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. +[Querying deltas](#querying-deltas) outlined the caveats of using `sum_over_time(...[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. This section assumes knowledge of [Extended range selectors semantics proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md) which introduces the `smoothed` and `anchored` modifers to range selectors, in particular for `rate()` and `increase()` for cumulative counters. @@ -355,7 +355,7 @@ Cons: Open questions and considerations: * While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. -* There are open questions on how to best calculate the rate or increase of delta metrics (see `rate()` behaviour for deltas below), and there is currently ongoing work with [extending range selectors for cumulative counters](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md), which should be taken into account for deltas too. +* There are open questions on how to best calculate the rate or increase of delta metrics (see [`rate()` behaviour for deltas](#rate-behaviour-for-deltas) below), and there is currently ongoing work with [extending range selectors for cumulative counters](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md), which should be taken into account for deltas too. * Once we start with overloading functions, users may ask for more of that e.g. should we change `sum_over_time()` to also allow calculating the increase of cumulative metrics rather than just summing samples together. Where would the line be in terms of which functions should be overloaded or not? One option would be to only allow `rate()` and `increase()` to be overloaded, as they are the most popular functions that would be used with counters. Function overloading could also technically work if OTEL deltas are ingested as Prometheus gauges and the `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on actual gauges (e.g. max cpu), not add any warnings, and produce nonsensical results. @@ -370,7 +370,7 @@ The current proposed solution would be: * no modifier - just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). * `anchored` - same as no modifer. In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas. To get the same output in the cumulative and delta cases given the same input to the initial instrumented counter, the delta case should use `sum_over_time()`. -* `smoothed` - Logic as described in Lookahead and lookbehind. +* `smoothed` - Logic as described in [Lookahead and lookbehind](#lookahead-and-lookbehind-of-range). For the no modifier case, the most consistent behaviour with to cumulative metrics would be do implement what's describe in Similar logic to cumulative case. This could result in fewer surprises if switching between delta and cumulative. However, the extrapolating behaviour does not work well for deltas in many cases, so it's unlikely we will go down that route. @@ -458,13 +458,13 @@ To work out the delta for all the cumulative native histograms in an range, the ### 1. Experimental feature flags for OTLP delta ingestion -Add the `--enable-feature=otlp-native-delta-ingestion` and `otlp-deltas-as-gauge` feature flags as described in Delta metric types to add appropiate types and flags to series on ingestion. +Add the `--enable-feature=otlp-native-delta-ingestion` and `otlp-deltas-as-gauge` feature flags as described in [Delta metric type](#delta-metric-type) to add appropiate types and flags to series on ingestion. Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any types to deltas. ### 2. Function warnings -Add function warnings when a function is used with series of wrong type or temporality as described in Function warnings. +Add function warnings when a function is used with series of wrong type or temporality as described in [Function warnings](#function-warnings). There are already warnings if `rate()`/`increase()` are used without the `__type__="counter"` label: https://github.com/prometheus/prometheus/pull/16632. From 62813e065ee8307a3c754e2e4a5bf2084b2e0427 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 18 Jul 2025 22:19:59 +0100 Subject: [PATCH 19/34] Fix typos Signed-off-by: Fiona Liao --- .../sum_over_time_range.png | Bin 33857 -> 0 bytes ...25-03-25_otel-delta-temporality-support.md | 34 +++++++++--------- 2 files changed, 17 insertions(+), 17 deletions(-) delete mode 100644 assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png diff --git a/assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png b/assets/2025-03-25_otel-delta-temporality-support/sum_over_time_range.png deleted file mode 100644 index 3d0367b588d7b753e9568095384043353a0c2e5a..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 33857 zcmcG#bzB_Fvp$TwJBz!!JM7{fAOs68OK=MkToy=z1SjYQ4-$eC+!J&Igy0$^2@b*K z9nLxT-tT+w`S-VF(+3s_v@isfpLuRl~;t;UFO);cKWX8z3Q}fsl}pMX@k| zGm&8ZP9!9pkIoQ?z6J!ssPFCJ=B3dBJnCU@fh3pwNLarSES^=@~0dU z=8^*vkBo`n*zay!U#2=;tii@;>}NQt*1gz1n_vd1XckQK#M_D*SaNkFl(kE3G3 zM<@$(lj^jBf-opIr#NZe$%E6%)V;|KHWY@N>i&446?`#Q6=%BT5EwI2$H}5y=lJ#O8uM1!lc#7iEZ-U#Iayu#YE%uGWDAF^7RVZ zq;P6Yvq+Of#apaXnmJna#jTT-#g!#F#U2`6ZEt9zU1l* zIkp838Fsbw`ZU}Ot;_{RfA#&Bi@1U9?VatJvQIkILqFUEUMg{ezrlm_rKTIFL#AI% zGoj~3Mxbahslbia;=q}`Hx;>Ex_-Jv`crPo$Cj1c z<;0uQykZN$*&(_v95ve>8^jAMU1 zrIPx=v?fsZgzf2@@#3@~t1gX~r<~`V zN09gEsm_w(Qt(pH(shus9L%zA`FV$K2X+YRE%es!YV1k?FAdKGBM{>c1|dEdtOk}1 zYlQi~_Eg0yGAhbbl}NC$GvBb^c(y^qUK^i>y<%Uc(abe7-D6~J<}&Z3DsK07GwWLp ze%90|>L~75&e+Q6_dEuPu#ubb%N1T%zDeFQQ`y?I=F@uK2G_=5CkNMokMEm4+39>d zYqE4$vb~)d?1wqkk8IWOeG^AK&vT32dm{5f=FQWnrMQlokoJ&-5C%vDaqnz*%Wk-z zr@zUj=loh~QJ!7XhR~2g-e|5t9)VPciFIvwv-L`(l9QWL(+T}{DCH-u_K^wx+ZwYP z*Y9TE%iN{iEt+rL`?OgcaZUtJ653+gKHnB!N5>WoX(oy5^-m4e_t%eTE%QuP4*&kx z+~l9)-*H}^R6T)HAZ}Ilq3eKRMLt9NRo{ENgsZvh>eIGWy)(k|56593Yg$GeD1J2R zElM+nJl-TO37QyYl(q5U%&Hm_22(auSIjyX0s{vTu&biqMn6i#bduR7|DIqz zPPZm?G91tsKOXN8FOZU(+Qgo~+Q$BbwQK9eVdaH~VX)!zGVNlFGJ5sfe2To8EM4^^ zJq5O8%3>UTd#F9{$PUF@tXDC&*@@W;!fGma?`}=?XV$AST+tgeKJkl_-80gE_WDvW zewjs+{vtSgQMjk>_eetK2P^s9?Y#35oKc+dmx;mEH)5BlJlQ zeqeH69c}L2IQ_2S-RTI%IIL2^^v>vV>O3KyrcHd#ZN8DZ*|fIR|JCn|CB#APGQR$<%SR>Lt?$d`J3f6>t#%9<>(|{M zH*Z^fF1BmD^7Pa7$;3^;!T!Jr&r^l z3Yq@yRXVe0pX4e+YVDkdqE3Xi4=k`c5Po0p*EhA6GspdfWhS0J?+m_rxsh)8g!)O& zhq+-X%y};{9x|uu|IVHKJjyE0t z*A##LtZXiacnRz+FI-EVb1+ne#)X(1tDij@uxlJ@)O(#TdbaSVfclWO4xZQ5u z@2#Dm9<0Q;62JIFu82V9U`ILJM?y+7K`Mt@>6%GSnd{+-o+t?FtSTI7-^=kmM-pv8 zCR)S@-+qSlOTRklnXq)J**h!=By)4EL01OZrFy-1jH`)c8kE&HdwR>+5n;pH3ip}q z_Z1PWzgAAWknq3ZiaFapJq3K&mkuTxjygI>+`usw5*jiH2^}~>1~z$Qn*TmlMdn08 z{p&mm5>l))652oa=mPtPPcpDQl=*v)ngT<@1pbi#TRx`r zdk+ul*yKw0&l;=n&SW~i^Pr?h~8zrR1fzX-pFx08U7l$4Z! zps;|jFduLSpHHB>uT21-yAR9XmHbyd$__qu-p-!B&K~ZJ5B1vEdieRuF*83j^xvPq zpVJ}0`F~n+_xYz=KnDdLt_TS63kv*q-9S;7XPE9$^VoT5)>2s&!Ycv>3$zHH)TL^8Fe3hTc;3edHQe|z@0|7Tt6N9(*j zGe7Sq)w|duH#X@RlR0QV^4NQtEoc+z0);|7M9KZ{4E5wTJMXSsBz{-bfBI%@Uj5_r z_GpN~y+4k4KS5&IvmoRwJLmbl^AgCV>X9)mo0T@EF)#A)z>@#U-e3~L!B+kY_wQ}~ z2cwf_l@9yW&CAD|rCOQ%HF^c|4o5*p1C&dZA3A?6zc`SS7$7rjiJ`Qd-D{ZAbr^t0 zxg;s(ZxWd`no#sz^-BM)+HlOWfu%6urT3*g`l(1vgiKui3>c&LS7#167nxR}b9pcJ zTjeL8KA#(K`}DIk+roOb)@HwH!QJPHg_jEZ(PINvHTuBf>zk8_5SF8=`f+Y4XQ%wY z(igXXzT+mYgkJA08JoL3uGOPy)LETSxc6;4=xm$#VRd&*=|3toWS_F;zZ^IORPV4~ z+ehGYGNBO0?q#gWDzW67`eFwj8|)3kqc-raqYpk*k_b3ljohF#wI}vpkLRG+ol?Ce z{#;2LdcNep|Gu)uTZ{K&^n&dxzGpbQJRjQE$ECJu3(D-0TTAtYD=r5 zDJ!Bqiwuv3y=_UDvu7rRN5Hyo&sPq1LbQ#K>oxqg2PY}u5%hDwAc^TpCgkPIy$E_^ z1COD2dG1njhW+Dfw^BFwGxK3CUu`#v7pP(H>S7px^efCqd_>XAWM74u+1yAIEBtvcn zHF>JZK79YyS2{|;maHaq^XI!2-GP5(G%SASF#(4+!s@Sar3A+H*QRb}vK}@*o1!M| zB*iNj91{D<+@s^$kT0M$ZP|`*xri&pf5#}&e|A^{;C=Cq2@LI%&eh|XbjMh-B6+*FaUKq&)!!o;tI5{QlVJi zVN;m93N>H-svIIuH2+3dRC06p1Fx?Z`z z+m^pQHIx|Pt-tWvtgNcjd>>T-sN+yhQO&1c@8$Q^7(%9pROUVFJ~zHN-FY-Ught*S zc7MHcM>{w!?AwDP$1l)vnsWUkRqrNjJ;(jiPuk1A{-9y5vIfX)9W8vVC`sPCNHyVQ z-dAGeP(e@4*p0Ex40Ha(>mi1)(+oz2&uWvl^;;9s@Q9YWOGQYQ=5kjRb*wQN(?UoPr5tup(OT3sGkXM6ECl}PM139 z_zOpf=m>vX)dbJ?SC`?jNgN*#E<3%JyFVTg zShBjmdZH%9HzZ}5i&gE56Hzt7WXmu?y!l3v+A`>8$r7#4=$f&G*FdDK@bpiG`^zMS zd=K6U+7+7Rg)ayLsUNg88mrU=9-WsdW!G-BLI&ft!i(M`d>bC(1U#4WM`LULxwQVj zXU4*EJPk$!iv-g|eBb4Ie-=|`(Y|mbvF`Pk9y4{<(k_PpS|KY>a7&E*pjf=4CL#NR+>V3MeC}r0*^^a_IUJr*vib+(!}R zWlsuTS_iT&>1Eo!ck>F8qN`ecwWT&Z3prN6od8bsM?QVf7d_q6J ze<<+qgblfL2tO#q<+j58PZ|PIGvWy%q7>w>N%=vQ+8eQqd#!Uev2?*jktf8MURRnd zQhcuMM}wWOb;RfFk0!QUVg^Mv8D%NqyXPxm=R*`e%p+oZms@j7J`-}cN1~r=+K)8_ zhz2ROs#rI8)u4(k#iOO@{htBFSiV1(sT&9SS$_RU*Zc;K7zDICG~d>Qn#Moko!h7~(CTd%9ful-dEklE#J70>o zd${Lv+?ujIii7WO&-W8$_76pPYIRL~#>8$(KH6UWEIaSYo{|nNWQGXfb+nj(O>$%D zgP=-!FtPU`_B{-sHI7`*NSU9-sW+BEJYz*+o>|vJ3U_NSc%22mB39cGi@CSjVj2@X zkvMXhCR9XF=-Su{SPG1af+bga$~<+*jEF1lfs~}U2tRc+5sJjN#U^jmH)67~@o?4B zuIV_NkHknRc{(!Ap|6QLWbnl&UJN;c3smhjabhEqzX;gYe%Jl&2gnW!Vs$lppbq&r zYE>*zewFJZuc7$}3>=4X5H}v5bzB5`<>BEb+wZviEsQsFA69}-#+Gp+ z6M#iyVNC41MW;4hz-q+NN37Viu&TJ7%|=h2?K2N?J?-kbCN#8k{7mgRr( zJY-}nKa^h$QHLgAGgtoYT`F!9B_j}0i#*5gOK|Y9FeF>wMd)^0q%J=}>BtdvsLCp%DCuF`zeP(^$Y0ey(owMo)Z@HQS`zaW1~BEhOPh=(A1ZM1hZMi}^%ytyPVgO7wl?uyY@3qKyT!Bw%O_7o=7SD<7i?p!^SL+q%^tCO1bgCiBCT33$ z@bfkDqR(<#Q1=)Nk;elDq0LU6h>2_l5F2l&r)$8h$+|yCVkR}`bS0b(8&8MI{_2qk zQ?oyY*I}pPs&aexC!30r4BKXKI??AMS_9QrQc=q7r609dTeqoq=Nl_mO4|h?sXA*f z|0J2##yGH7(;n|0@iRtmU($VLP{t)CxDY>aD=7;1)>9|lj!zX8M;Dnsc&?>d(O^A>ICv0{M{Lv;ZmFeyo)&M@%_>BYCWu^3r$dGv*Tr)F2`3 ztaQ71*9eTS9p}rxC90WJlGqYhK%}SKdo3G@&jZ@~3H$xNKizJ2)<=A1w&xO!u78A> z`gFOKbL|ox*1JXt=Qx?kiuQtbcw?o_cVoxWnErB(o}%JlJhl0#2t0;$buHj;UfL1; z;jI4>uBZmd4JSuS4*{EGuQ=wo(cbrUn>+)pq2-(BTjAP3*mhM0u24qMK{9C-_Jv1! zsIQ4EMYP~dr;!scdTCW*&OKGMk0Qd80|8g_>W@AYYX$pG?Y4sZ!J*5DhIQTUc`Tn3 ztQhuh6%*3+W5AXfS#9^i%eA4^w3CcZP_#=M$;VKk&;ITQ9p`7@=bTilZP9DVSWDXl zVWGT^FbSfsz)H}mjW&lj>_bL$2%0lAC9;E=W(y*8h_DfXs+8CnsV=eItTspO6nESlNUj2TZG{#pueo z+$VT;l7Lrn%Vgsqzx$1FqIrHIDI@A=rS1R7;W<`Ch*h)bMJZpQIKOt-H1m zLXxQDA8AS{2M-a8tW9bOBvhaD+5|dND1Z5VLN~z0O+jd~X5&oRNm@q+s&y?EiAZ*Q zY~zd@Ykg#CV`9Rky&KH;Z&Ka*!CQqCoxX1ihRl}pqvLeq5U~R4JJ_Z4Q4=I5b(%4G zqEwWZ&l1d%$qBY~#K;)mBXs{NS_RE2? z+dQ?xGz?v#SV!5F8N2vC$lznrm)Q8C^AP01MvK_K!roz72MM<=Qpb0za*Ew%D34NZ zzS3y2#GJCxiPRO$2@o;YzO02CeQDTa6GddzQ$xprfYMRK!bBRJdP3vT{xzpH2SN0b z@OgX2HxCoNEWZPO~mKHa|LS7ZM-k*Py3pM1mH}*2Ju#} z8oyi0WRrE#b?S|3_*Q5euuSVd_@YD+7TD_0&*UC{D$8%c5EHL5e>N%AMmFyf(f1W( zOmTuWzF*ifPYJ4Acb`qZkj=out0=6~D71}fM02b*!B1{NNPs2j59PPBj(e^wIuFsT)5uO0gxw22^Y!g0NV=2EA?pT28D<212junIf}?9Q<{p1C)y?*!b)RrGV_n1rk2*qH`;U=%P^n@< z835Ca2xDE=h~8^j7Cf4ItGJ|s6<#}u*B_ex!gR+rNHZ(1)L=Yh8UgK zW=t(CPnNq_jS4lB;g&xcm-(PjJgw>nr5Gc^)$YXp@;hdRaDZ1FZ1ZE*F6qxYx|ahm z;ghp-#~PicFu4ZD+;M0+q`cN!$kSp>0_+FQBwFFvgpF3&py)u)z;mJVagA!Ne$GJe z*#&_OXR;|d+XqVV-de0~p~}FeX3?15IGvmk4^td`_|R=Jd1eW!72l|81eBT}H5p+Y z{XQ&PO$x8z2;n8rpv~)OOqqY%peW_YNe5xinxXhuGwn@fNwZxAgoelCiBmOBosVm+ z*JLUk!ibx6{LzDw05OZnL(G!luVTgL z%F4>KG?UCXLqw>P3}L{B+i@p;^n$}LIU%IbeF%Hq1(0 z&?J)y#O=BIlDJM!*UUt-Oef0$*8y;$1(1l&4h5QKZ9}t&FaeDk!pjj>HvK3(beVSG zIjl7pA%Yf9bY*oU^G>1o&CjPt3yN4{9ra`&s{IHtQVM<1ACv7%u^oK?Se59c{+3l# z5iSm<-}D^K66)kt88H}P^2&~(by$L-m4>WnpQ>Mr zljw2I#)4Hcz^LDCpyXg_-S&Md?X$0-)f?ev+?4T=6Y*(J=HCeu4?!6mb`CD;=LjjA1&`j z_fTIU>d#$FT*(a5bcL~OrY9t8rP>D#jfkIwI;PH$#7 zvHARmD@J3%NZ+yZGU{}&Xlz_XUZdEceta{CfzQK(dE+k6U_!XT@w6lw08|+BKTQTV z%?A3fgfzEpB+A^dzuP8Oqg{jQdwCF9?t(;5cS5$PBBL>q;Ouul9uf4-v0c<0@;klP z1R)-fLAxzXlobisW%b8FP-xEa$caa%7WfSiN>kx{mtWsWx}^N*ma#Pv;n&IXfE!RQ zwa910V!2U_Zlf-k;^M>W9fv^&?7Z&-IFczl`xZICoa1SI{RLGDf8(ok4>)WryygaX zlS%Y1oB#j^1U|PvKK0U9v4c5iG5u7o$yF7LRFRBAO(%EZoE58R*Pp91r#H{5fNU&+kseX?4#$Jgs|+S8R~K!x;!Ev9ZAi|(|fS$or^JVD83&^!70}Z z_F^MJVz+GCVYX#0+?)HvD)=bIqsqKr^j&SVCWdx_+q_n2k(W~KAQ}Zm6-#imTm{e- zte04Hj&f_HZ!RG7L}%);=-BwG@2)Zf(;(%m;X*f^G5CL89CUnNdcNSs*W2fWy>Od?h6) zz%n|-Af(^jzHQ1kl1a`lkUN|0Ggq3G&&bF?Ew{CaQR*P3`RYiR~v`2nYd%|!Q%T59Af3Uoq zqd?vznAXQCtM@xk%X)YAV~xj3r#!KiKt}^Zh8 zU9-YqRoY*zr2uGn*J()9E6tFGz+S<6v-_lT#>{1ozpGeDxi8LoO6DqoIm{JT1kXV{14jTW+JG48VT6Y_oi`mR}8Pcj58BJ zLZ9&Z&K}g-Vc_$M^oBFWl_+yYMt8Y%1Y!MnOrUUc2vN}P1zi(i#}e5#MiEV%bKs>R z_se&tW4Tid^TondY@l%aqq#|5f9)Ujw@P+^u?E0ef4qG1h6b72>~t5QI9L=s$$&Z} zWRH?+P?};ddp>Qb=T_!F>wUWObs71Ion4Xtpt_k158|SyH{!lHYK_(zziC{vpvHU- zZKt9P8REg&Nu_;*YdDsit0#Zij~kgL#eL%JAR^20+iw|Pb|Xy=7O;9#Q#3`i^)r?s z^icbg|H1y$RQ5r*;8U{?mMB&sf6~KB12e#D-QLkVgUL+6u)~`Zh3fYCqd;C_bI#b{ z*gz}BLO)+nZ}5?&VwU1fxY{I%kgW0Vzb!Rg1F%$wWy%uGhalm9ecz`5F~j?IH_e8B z;Zuy*fp{)G&?=4luPOI04gY)T0)ERi0abs0+{E?uUkDfdY(R=eTOp6$|F8BQ{uXir zZb13)LaFkr|I4~4I*=qI^s~=ZDoFnS?L9Oc@t&0t4(ImBApubF|3fMLKklfX>$_M2 zx?i2w`!dY{cGwE2wCvO?y)3~VF;P)_kW;nF8Q>+gN-PKT`Td+6`15nBY^JX4MwB=J z040_H5c)^w#X7Fl_*SluZu4j6S1mtFb87miZZEeB5|4FTE!E?%C zIs0F3W?r`lwC&c7%I{-QK6&bwZf+aTwg^D-Z?xuJ*d_sN{EOTv0wOO1f#Fr8yK*t; zpvhnFiueO9Nk+pOYX8ibzldn&{}&Ck zwCFRjHx5usX2WJrw>ANg**nw1JN|k#itz0w1M%v`Mv}jwj@aY+mHD;}0O|HsxVtF2 zr=SN|hP{Lu$BD4b^e2l_+qoMj{<|NCKCxw5)H$BKfBnEgDpmssl0J{WaBOfi!}E6& znouec$JYx^1tE(UM;lAPf_%_4|E!YMz)I$k=3o6E{UXqHvwnBh+|{}gcE5Bl3h-e8 zJSb;&OV&0^e&79o<)PVat0@hkFKvd6UsC`6l!YrSmhO4h#wCzNIwtDgdAa3N5PDV7 zyk*Cbp;A^5`Y9*$>gU`juR4?y-@%084MHCzLdW)?w0Sy!IXbOG;IzwcO`YZfK-@jQ z`cq?Y--oWdUuR4HRt(M2FqHlZ8WNDf$LW6EApL?D`%(p&_!5hVy-&Yu#(61s_!Iyr zllC3z8Za06;qumoTTT2Ve zbVHwtJ~UkKx5-4y4PF+6-CcTZ3{FOP5ClEKUT|-7+n1MS6{WhI$s_@pIT_)rttO%2 z%2a`B&8*9*Y-j*ey;7fqYVphzgT~@ z4}^Uu8!w#RIs3(xtThv?vwhQDlZ!6rrE^FGi49w&dM9j*mSmU>XX@}r zZjxE3lYU>5iy)Os#Elxz$Y!#?uG$DASQxYFN#o z82sWfO??sCTleC8KlJmkDvcZ2kn>M273gZtm_+))>g-3mkiZQZJrMcdE0uKq1Bm8i z`SpoCtt7*aThrX&*NYzPwK0j6-6;!gXyLl|R~1U~vG0R49=l^YLf8rbPZAQt94MHM zoa@#|0R-S=xC~$HD$pj?apbxUd)k1QxSiwn$5E3G%T9Sw;xWGXg8vYme+xiyFFbi( zUYs>++f`poqVo1o>A}JQBqic8=vmr+Wn?L*b#Z>#ZW1EXxm{g?k$zgbu}A2 zLUjo+lixfao(H$!&Bb0fE?=Srh&#Vt_X?eNF4M_-iXyak5O&|D$D4lqK*@i&KPk9B zO?U1Wi$+1KwJ&^^J9|KzWNj15X*kB3`Wdd_0CmId0Kco7%dIXbfd&%Fr1zfKART4e()Czk;B@F>)LO|JhScmc~McUq`wswc6TZN+#vF9u+mmmN?s2~Hzp=ot9pF_mJ^ ztN!5`hJch1gjZGNx~7pe)YXKLRZf%-8{b6WfxwXW$b}pzq}M!fA?HDe1-(_Opa3e& z5sy=5k{u;@kFE!E$0Z*+x8`eS4nzPPv()8#O{Wc6&@O|Zwc;WacY-Ozgc`B&Dqd19 zbB2g%42z3Go2=F?<20H%1F_&7p-C}AZ1j$ZHBN$z?tzzF85uru_7bMgTILIi*hyeQ z2=-g#V_?Ix5Jbk|Qi;ox4jHC{Nk7Xtz73vD?evP8z^|~PT)$ z9p_H5s4W#RHhdFff^~&I|NG(f7X-0S0QF0h=lmsv9g3o}Xvte^5_%pK-G{P5=lz^oS$Gpha70In(spd__8g>aV$&8RUTiNm+!3VFN^Ffss{pfe z==1)Bh8(ossQ7`bLmsb)F*>2VmQwn4s_g2b!0u)T5dSa9oiA=|SPxGCDH2~lP?Can zdZ^`tTNE%jB8|3q5eU{X;|O~*L0B)R5n;rk)Ou@4ifbZs-=I)>Xubl?9pth%vRhd+ zh_u3uCMy)@oTgnLAoo|z0o%VTG}cL`%KmVYjq!&Yw2 zyYBxmQ?6&|eEA%j&(OA#rU&H17!&o+y7M(`=FpU;`lIGy+hBFcT_h+Waj>dHJcPJz zwLKVj5-=MeIaGl{>*Hk)#8JN+T&OO!cU20epm zGI)4W)|?n}mEs3}<}nySmle_aaJ!bs$IupfxkezpsEm@N9N{GwZclW75YiLsn$TvPqHz|Y>;aA%udv(XlsE;DL zlW|Y7->{2Dv+oV$`cI}N=xeHY$1E(xa_3t1e_6e+`1n3wKje0%fp`9 zP)-~~Q3~z0E=w|?xNx=^mtX$MVcKzpUJf&aHo5S?_j!HZ`m0gB>pFn>g*4%?bm+~D zkPQwgDOOg$+-)V`d;#PoK8x@ojW{jHfHccs28*Y6o_H~vg>x%vk}F=iypEH} zj);C|`4$nLEq9NN_1ck%d@x2GI`b3=>(V_6V*rTb1l!s>=`rq|_71)T1E=UTaqHMR z8yN`9Lp>59XAbvt?V6jU70l!o1&qn%J5Hu(H*wKlhzK5v7B=`o8}<%qDu}r$n(2l; zc12HS5Dirssh9X4O1;=rGFAa>}#60Yx#bN~UEI)KqUImw6VANOqT_{Q~>Ykgdu zZ8c}DHAz9P0hxK<=Y{$5s`H9gWbzdyciB%Rh;Z;1uPr-*$XUnphYn@*Jrx_2vWxN3 zxkWugTURW`m7vP~{d!n^Ah*vLT`|BqoGk8(_M3y0h-#3`jGBZalIuI(hz_S{RveLK z!5@9?k8oONFa+9|`MPLU9D+EUD*hXuNLBzKluMDCUp&$HE4E-%>f+-_4G!|GhVY-v zeUw;kl$YYtbnUhVF()!bvfCe-8m5o{s$@$;YQmsU;~DH z#0b^`IHjL;7p~LrLL+sun;1#6$%vM6lUHO&;+|+oYg2M z>d9!B=dMU6?n@~vvC43R7>t?(Q{nnHYmFX6oPGBs_6R*G@`BleD8+UsL5IL9`Gp$2 zSJq^L_gb4(n-moA9g)l}7TxKhDW^8%CyRVTtnY{gkNTz$E$kUt`*1qQzY z!f_bUBLmOX*Iwl0AfV&<3*ONO0m3WsdcRPlNe;pz0v2`D*9-l>+XM5q{43)c)u6S% zYl|nNom^^EScW4Lu8W7rrnI$C$)!d=UD9nEz25BS&49^9;NT7sfnUL@VldVP96P_9 zT4RZ$ZKMNp;MO0}04n)+`gv$X6Xl*cQ_3J7*;jDV9{X`K7tb<4S*-}U_l;Ilk>4Qk z-~j1B5Gq1j7KfBNjfrA(AJ6%G?>V#11s2iS7k!g7{LaO11VAc_JYc}g&VohAt-ad= zsLEz|qL5ZF^9^BBM|aJp1Su8uYo4N`W60iGVASmm7EC2tIKE4JgLR+xFgb{~#-Ov+ z*j*CSi)8y%ar^;e9|6RLgQtv^#*E@Z%iUTiS+6qVGME0~k874#c{=qcn(BYg>hSG~s zqi&NUPu1ifUi161C;@iOT#vkrRg-eYNFMP%p7G=ydKb|$I$1hv6qNS(gh3xj1!;xo z7h4E7y4Cp$`*uSUe3j#xGDc>Om+0Zh5tv@xj%R{r?S@(O8Job;MJ?e9=;~pJ^MTB( zp$tL03|dgbDeH;6;f^edcJrwJOQN%_Sg!Tu6IbG1CUHPd#nWI;xIOcZklL1bN*lu8 zu?8_P@2*yDDImbQzS3HMO#+j)6uvY;Mv^UW-QTu(HK?aqbH$nD=xlUh3)O|2^o6)mRh@M zkHMfd6&^F8I+R0ZLCK)pn&sAkV$mqzz3NsLn6u*)(bZUC53CTZP6`4Wy2Biq%2`7Uqt6-LRB0Gm?|i8u4i~^t6kd$Z@sTTJ z`KVw?K35lDN)b|vlFW_FY5{PSy8uqz3NQqi7LR+Sq8pfKgSE1@=3sI#k;>>z8Nyc> zF#2OY!U4HuYOHs~JPq_*fcK9eJ&r{+KBmGrj!_|rkcN1Rzdbam{u}SL*8&i*CRrE$ zCIC!8lp%Gno$9Zy|5neZ|w=*cO?WU5Oj`Aa3dAq~T9#o|K$ zzc6xwGO9$yT2;SUpJBgp*HsH*E!-*$PfK!XfV+D;Vh&}@8{NkLu)oySCb1Gc^~o2g5pDED40@)9nM0Y=vMFrqxu__y zVax~&*U41ftqg`WxoRK*XngYmRkbUfu$e2Az zTvqh}Wk{%%c@3I)M-F`ddzRo4P1e=^m`#w;eiRyo8*v<`ZjIvf7SZjs_TuF^=>a(! z9Kfk&WJ((0kJ5a2-#>7bx`sQ-C~yaw^Vj}d#tvyZj>JN5kl)ZS_4aoI^XKru0~r{S zl`L09>G@hy(H@WPG=p0-?Rw13)%EqQmgm!9A_scp*G#0M?U(dsSk=#uDL^gk5J@7e z)r5u7y#=mp?x+kVX#j;VDPF+HSb4cp3jH=T$dru0wn1eE~h;XPH8 z1>%Q!1zNUl+rcydjka8tf?-e){NRm-pu1XmYdAz4Rm}))_;XXtWHq@3d3e?hdeM5M znA#wDF2!)tDQvS!SIL=1nK_*R_8bC%A z!YryS3(@VTnn?T8b1{RW_t?Hyao%gKvliLjU|tXNfa?}2&`EC6yT^$CX3@` zMMbQ#uYByHhnA38+!YTSizfH_h}ou$JhgDAl!1j2P8q6w`W}JE@Zg_`W54j?BOT$@`H}9|S+YhHok#>yCuF)D zS8UjwDx2tMDC$~!!OoSt0zik$)Na9^!#I%`5Yp5>RRe-={D0HbuK`r`GnRd|D(TzM zQk@g^&Y#&1y4ST5(*f2Rii9>MQX9>LMfs1l>HAc}gfQbND7AtFcHo65X?RAgf(LkI zR1c};ivUeUJQ3x-t^f5MQzJk%Us{s(V^nc}W?2K$u`}C%>HIe6t)T=j!Qmv`+vitN zId`kk#{dj{97#x$3=1-mktrqN3dPk?rdyf{=5^B2C~Q zKwo@^arL1TT6Kk@uTX?Km6C&6(Rj~ZirG?Is@o}$2}4a-x(%NbVO*+ChV!!lcEILw zCJQ@C#i;Vnn#?#|kO*Gwj<7Vs4QVRI*Rr06<}6FO1x<_ybS=sO`Dm82f+L0c?W=Ts z+A@wkiu3BEeTFIH&psyGw@~)LY_1YY6-*pF<(Bzu80Gt))2=d?bN-X_NDJ5o0FG~c zP}ai*BeD0MTC9Iya6?x({g*-K?d zh|X~i>rE6bv}L&LP<6K|0h}XY{7FRo zISx2_t;h{}u$yv}2Xb+hoh!ak30N%L7S%T2H9hbX)z=k^M&f|T(?i12q#thm{J>!H zI&qL!G*#kqW0W+FVRnLz`Hkorvy%_LuN2it zy63nFXt0AZW#*|J)V3afe;_I&f*$;Y1AjDeqGsWOqQr9&vm1$H#8)^%qG;6%LedO4b0->}&4& zQN@o{TyqoP6ZgkuPe$Ksd?4XQF(s7*pz9@oGD-7mXdRHS|4if$FqPXBUCa~_n(TWs+dcMW)|W#o%@2C!5AbouI*|Z- zDE#ogj*(7Kx<2vex82VwtG5Vb54)?1#`rn`@@C~0;7M8sKE$aMPGaj@ay9=I`s;rI z2Ejhq|KTkI7$HNxSJg{hwD+%&C{O^NLGLlESwhD979CCdM6}XuN6-V`_#+@@_ubuL zAK}u+A^M!B(JwX7-X|+I#z#GlGtr;`)FK=Z2_7Tt5pB>8{DDb5Ys)Zu#^rzdRV`M* zaT>aHeXlI(F@8-P1)*yjUp8o4L_f9-vBP}JZ1uOcESs|YI6B_J(`G)sy~ zOLs_1NjHc=N_UrlbV;WoA>BwfOLr}K&l}_8?{`0Q=l*r)&b{9mW@l%2d7pmHbDrn* zdL7PruTsEZC>GV~_$IK?bw`5G_*UcQ5BV}7sq<0{A>TYOl~DUBZi zL6T|H9FVTWwg;$|(Z%!xVgqw1tHHgJFI1(TK?jg!mJkp)mZyE0IHgO}52p+ek4B-G z7Geu<2d#bS3!g$i^uqLl-2ucUD!xYdWPs zUlj(NJ3>F}YyB+qM}W;|3HTq;QnmoZIR+BXI?7!#s4t`W4^~2+JCjY}JQIMH1Z^19 z1?!k$ByiL_hkL8bY{P~WBYohP;dM~<6{KZrIo$eLOuRhqz!ITtt{?Q>4ih1!!U%iz$^&c@U_TnvI#rirxIA zi~_tqTlc$Vn~emO0DqK|yd86k6<6K*x!qR`YB}rZFpiteK6M~1IKKwq@ElyuY+AgZ z&QE(`qQEYC*v*4(vja0zig3oFi3ky{^uqgm%C@2&?k*!l0E4awf?HRDeXeS{sFl@0 zJmLx9m)^&^dLnsp(;c63qx%U-IY^jNb&P?-*J9*Sad15#)~N+Hf~l5U$1YBpu5N_m zzK4C6%lcC#NNdg*Ugw;}D(w-aYZy_1$XD*76j1xQg9+Cq%ohPwC1UFKtuS;s#*6g! zHGpiCPzR6_qv`0K) zAA?2_`O#hLpSB+8$w~#bkl*sMeutfztdCQb*Toe<>@^w%!BgtJ8Np z&U`he@1pywr5r}}id${-(iubR0Hk#U*m2HV`XGjH2QtPAFJ;0QZ#IY4fd#SamUs;z)Q3^?tdUF5~~Dk9yi7ZLe9dvB{PsAdZlVsuE(v@zX*jQ z14*EU7V?56Jqp;N*i9U-Ci&R`_^k0~=hVfH1JsLTYy&Xq;~>LMF`oPFwzyPN0Lgj? z+n}0NImE2R4#{Bk=ATE!CmJ=|HuF$c|o>=u+c!d!2L&fOYLtHm6RS-BN05|8@jes>ATqpX4S z@P{IM0J|vziI2vSu(CIc8SfqDqnhHt%da9J7ywx03>`Bl*XX6S7_iQy1?T|k!%+Zl z4DgXh0j*6*fJRP6n3NC2TK7xv!ov#}Zbd^{0NqJ?7lSSq2>fkG+n}ttD}d~9Gy<>& zkH(-6dLNhjOXpJ|ZCxLlv@Yw<#ixO^ZkJIrSCLJ5-vIvT79vS8Pvbgu?oZp<$p^2S zy^>A5Rt08uKXfMF9p_F}`xp0x)*p0%PbH;j;J<;yn81MbJFSOW{u@Y)1Nd>YYvK0= zeukO<%d2!XSWI&3)v_%=LoJ}Q|M>$JAGmvEs>!JTA}heZ3&PdX)%%h(xc>qY^8xlK zY1+&i``^g^|4%g3T>;}5nQk@HxalU?+f$MN(FvPX9$M)Fxw=Yq z?sdqQ)&yg)v_DRV_`CNXJ=OBI!5@p!lx$`&sLbh}G^GTWqk0Sx*QYel$&&D+_qJG9LoQR2XS9o zodUBODy209!yltdkjdY|;LaJI^iaz=d}A|nGT>@g0)h+gqnJ`jTM5=zjIdTTO|*b=PO)KYQ{14O zZJczR$E4)1dKq-oCuFy8;%pj*YZ!X@wdwC ze2U=c*;UFZPHS$++dXj5I}zY?XCE0VcGHjo>bil6+-}}O5m%-G8(i~O0pm1DNq(I% zk~kNoCg(>QE7YYrK!a@U?bv5N0JV>1Z+>D_5Tfc>9&8}8$baX-vlcgn3 zmB-4|s9Sr=CCal!D{v%zR!I7@pT~yMf5+|=D)H%}fueUw zSM)XZGU_#e@?#T^BSpFBK0=YC?HT*tgUQpzuBeSXiL5SH{$}$B{^5P*qUIE@j^R?@ zqb6-^{6gK#vL)X-71c)-)Dxx}Bpp6)oWu1V%WQh_uVO>xZO0Leo#e5)75KZQcpBd? zYVXdMyT|&0p$eJB_+wH8GptMc>0{MZvW*dQoZ~)!@f*0OO-09{sHnvg}ro0dmWLRm2aeV63TNJ6G1j@ zt`N>ZloM+(7s7G86Jo0|T&SeBd5ZDg;@eSNKN4EW*MX8xTgcD29`(nHD`_25;#-DeOu-|N~8LyXMQ+{DU7XuS}B3PgB z{t7edOw$Z_A-Jw!O5PeGtVgDq_HOfmx)3@`^nka ztO|X~nNG|pZp=0Kc&E-SqUAGOxXf^icO7+rwVhQ*P+k}RTwe=iXF}MGGjXJWr*4a7ieChf0DitdS`mKG-A{Gb6!bTDr zd}X$Til6fm+dP|uK_uymYm2Nsyy=I3_P zPEHRDVa8c zh6d9wZ|~JuOLh^67BTwhL^TtR(AJ*T-&OB`d$@^ca=C7wr38@pb(aa?o8q3JFI`fy zum4PmvU)H7M>Z_i_#UGSS&8M#uzn?apB7ymS<2#|L9{cPbkq_3jIG6>g#dKxTkaX@ zmS$fJ2O{$YmTD`XJA7U@E%2b8P{WU7V1s}6YO#GXCaIo|hS6-G3>%lzdxpw#ZYxWi zSSu<;WSs6}>S(M@IfgE#C22=BCnps=AU6FkehEJ>PGD!DW6e=ZC3>hx;`}?0)%};& zc07Y>h6lS1Yqon;l`r-!VuQh=iF#Aj zrsuRq8zZOq&k1c!b9HKm3TcR96%R(fZAY&sgxb0~O{H$;p1q&PO!7!z(#vs1=`QMj z5p}OIO~<|?M{SrznUN;sbjaU7HG8XSIC%S7!qFsqybbsGw3cT6L{nHH6C;iUV#I2^pm5Z8@I;?ZnV$=53s(nHLoulF-y3+RTN1L(U>&0KUsfjB$Y=hEum5;;D zoCkaHx*z_?ruRIGZo$?oeezkdeB};(rO4$#7oKfbJl@h0(+OFCGxI})>eNq|kcjOy z&3($id^e(=;`MbVCl;(Y#L>h`RodJ~4(hP14IbaTA$a8-qpLIbYB%?U`GkxH=kA`t zjasa>?z)aJFWD0u%4{FLQV5t7-DWQ{;BMx0R4`KD0iG9;7&I$tuCP|b#J@a!aWWOP z1|l}e?Dlv`O;_E@8Yg5Vy)-u9SaBx?SC*aIk}1zZU_IU>T?;~YPUiVMoXe7J;8+%y z!glYwrcLjV_@_}WUv4TiJ!9}s3kV^?JtHPEAdw|W=gXrZvh{Ch=*f-SBG^0#NaRa% z&>MsYYCAFzg}Bo3nD?AM8;rM>bbMSoN!ThSI~;Sy79?LTWKfoR6^UhU{ez>__3Ybd zfrA>YqsI&jK5tP;$HVyS(P(=>)Ggk^Q_#w>G}g*$Q)fy&R$g;k?3(c^30esjqupjd?+M# zkv5HCnM-)qVZ_zQSL=N;Wq#^p(j1oGJ2MO3x+5tnZcPkpu z#2O`iFG`v(0o(vh8hC4==*ZP=)-qh;;KxD=JhubT`^X_TXK!U<0S#u%$m-0VTp6S7 zvCyg{61kP)c z4dbH=Nl{hvO_z$YZAm(-%un<$obu=?84&FQJ7iZSq41zwp0TJ zO7T-@MpfT1vmWFfIBqy>mmO2j)0o46_3oY4pX`WDnI`c0%$(e3gi0vEvA6ujQ99s? zyb8-39d2i+4t?iQGC>(fgVF#M?n|P6%H(M_G|9*#J2AZVeG&6!xobFu|h4DMfuOH}6r)tj1H$FHRn7ksv z9P}*?^eGWe*%?h7;nXZ?Gup>_t=pMjwyT|H77j!Gr6*_el|1sCn+ zF!+W0W4q6z+79J)!|=MR;G@UQC9BGoduaQIDYPl3Dd|PRiEwDc?^@pwv1>Sc)bIiPLID>v;IQxsbqkl~?GZu(VM4{UHI)}p~H3yg`)v&r(Hnb8)fSvpo8Hyy=6C%`te zLB6KxEyszZz{ZDnJ~vz_wRlVLss^9AeB!BkqCUoog;q-Cl=tycy>DC3P9@f|SG1BeuQzb@e(rEN$p+Dgc;3g)n@~k+I688EuxlnwMJ-oWged}g zN>e~oVLXLPa8Wf}ZK_nw(!sGNJ$G?QLG!Vo zkUEcai>@b*l3}~Z{)G9K@wb&?v$vIdG9d5fs$=sdn(Wn?kj|KIp7^Vf$M7I(U-)}< z?AF|pJTalA_0!m9Pe=PT-H=D$HpK~Ss*LPkwDT~wc%jbl zAqb6Jug<85{1qC}Jztbhh)ED0d@@sZoFTId1*Xoum6OZE&6Y|jeXQ0IJ4_85oY|`S@`2MfHm+Q)Lku~R_0&6 z`y>Sg4Yq6O2LBEk{CXdF4Lr46Oa9}(d?%d%f;xQ&7scFv3F?qS;&PT>GN1m-cZqO_ z71(){tM+fKKt}M?**Ee*P~i8!{qeg%!JveF_45DX75(y!)S!e1kcYVaSHpk%O@|Ph z5WZdsY3%(Da{PHy4oGBy*AyPp{>yi}5T(*(+-^?guZY#}=#?du4#3U!k^7hLmY_u4 zVbyv|@&D9}*I-J5r!p$t&iI$_{_h*evl|yKoSlVqaVlQ8KsEzKDOblZ`b>4E_%;xy zfn@K*$)Z>&dwl5epN6yGhGD7)wFLIbkjXw14Dk^W%|?^KA0Vfc^+-)_yNQQg&VuW6 ze6)6Ybi4<+ZtE@+`=tRL<@?*&HtU=qjvl|g(mB3XSc||*<2_lIxrKs?jz!A<`ohIu zZ}_jF^rFHT?q|OF^U=>YpHNX@jHv(Z=ZTLl3J4j8(vtu61nB3eu-xAd{Qa2!{PaIP z@Sh&|PY?Y6*aJ9NWq|xioj=%Trwkw{E#$uREW1~ogMZt`*B9I|Fn9nPd~V!nJ+7-p zcFZVDUV)}JO(JHg>B*_p8;>Bq_t&+$P=MHi^$@^9TseHyn=Zu!GUQ19TA{hWY1*Cz|4vFg3$`iFCJN-*w?_Vcs-AG7`1<^E2aI zfblQtEVsV*Po+>{Z)?%O3;Q07g0YLquzBx$$GwsDhg5F~H+_Y-F@9G?L?lrYoJG$1 z(=D~A-hN0Ee9f=veoWnC-0#hHMuyG0Gq;cSXIpVsxQq(RvE0iR5AcYmXTv_wyeLqA zreZJhr(#-t;1_TC-oEkpLU8196Bmp1!vRL0^FJ+vfz9iZ4!_i8*rhk}t1A>6;;1k& zhGIEolY$0)64obz|2*pHWnv%`X$cT1ZG9DNo+2?RWYJeIqPSd+_fAY|ck4SO_G15Q zBfWoeI#D^Xml248pPYe3+NDT(=8fsd(#&@DeA$U~{I3m{ziR_i?UIKi&!y&?7oS0e zQ(R6z$N!la{IZD?0nR5S?0nqa2uD_7Su*~_yw{0IOn!eA_aT(Dfi} zxzS*r)ZKvG@B!W|p|t>~-0vp4pJl%OPI{%&eBs-qpG&>hH69#+`klqzR)C9yPs{u@ zs9Fb?>CHJ0tnL}!ttv3TYXSP!D}leA`p-V!g@wh+`T%W5uImA*Qhx^=Op76e8W;BO zqAr3DzGz#YaQU8XkJW-}T@q76i8cj5PS*z!<#zZ_V{;RETZkPTU0` z1R8fV?5EsaJzM@vXLPLla{&HB^H9I-?)?TIy4)XD#pUj6+q-R-Eu>^v~a527&eGCFzm&dzt)ft^0FmLLcJwx&G7T&;uU`gC>%3;YI(`rt`*$ zsDoB58Au=fRX;Ef>##7;@jVuF4Z&>wyFPbqKu>;35Lo@E|IY_Z=pN|P|3j63i(?OT zm0JN=MNz*k@;T68Un`sq77GV59ogbBY^JX?GxdK}x{mNZsbS#XgMrao*Py^cz4kBuvU>Np|jVId9$J z7DNp}CY&J?f=?X7cFgRTRVQ9Peau8S@_WMKfK?=Z;XC8`z;U5?5?Vej2e_an)`JEn zSCC?AWoQNSM(K`hq$RMdX$Po=!U zkW$U-GC?2Qz32|1IAQ^0BrZG+@s)c5*|C+WSe;pvGZ=zd00Ekss%*a9d=e@TL`vG& zn0{9cRi6R$XmJoP^nn&CDnwo=1S}ALqeW`?rg!1-R%^h}zE${;?8~FxvIp?DaDZA#Em{Uz z21)|}7t*mzAApRRI8Jk{u0y#Ns#voX7PnuK0Fl+_fben-8&hF3`HUn>R6lcW`F!*i zkRL9<-`fPl*=qdx0zi(AC>~JBqA%`~8SA&xNUwG~s3*xB#3K);3TVbJiM116QH_uI; z&SkcM4N^$WVwz*by&ZpcVCI#ulgaJ->Q4wb+JHUu`EN{lM^2Ka#3+Bx-RZXOG(li@Ih1hl^k`jECjF=R z)V~7E;L58zM3;bhoPXtBdpyT9=d$jk`Gkx;x%^9Jlh6JX%5 z?2uJo2EgZp;0IIy@G=LKi1Iv``|@G}NRB73aI-g!RdYoj?K+MVvGyI^`=kU!@3XX$ zhUzJ{4b@zB`*+Rs0Gy5v#PW>#OWeTZ>)(XbwwZvzQ#VL}K$V0}D+O_b4xl%d*jd`h zs|B!8W)GMEA0^m0mC;DHV_(gxUEVy?Bhq<6h-9g%BZR$b3MQW)lthYm8d%O#@is_9 zKxn^sXA}W1tpWnWc_`tStxwef6J)SyN1qc&$B21k-ap?MwL&59%z5u~9JHut-PgPc zt$LA#5!EK5TqHbXCFlKbxKBBjXosM7V8*B*Yq>y0;ZYloX?p!|r zsy3SK%F?C>`>ju@Lr6YmT#+p98N^rr2;TV21)2fOiR^maK?u=&JuJF1df-s zrf#^EMF%9h@@6%G8h?EYAzblmf(Uj#Q<}EKUd7vziE!n`+&WF?J6ZG>(m&ML(SsV{NyKo;bJ`}3Q?^=OIek%V@b1-d}9wZ%1! zh~wcn&FXZSXl+4wLQK75Ft}r_JSky`5Z0cAf%Mm4FB*#9f;LD@#3#h7=FzKBI!TP6 z3&OWY4<>N5_h=UzApxfv$E=~O#By`T=f|_ud>L$%7m!;KwLBRy<76N9^Nod5jh&HMyk- zW*l*7jjWn1UB|bCL8d2Xypvgv*xwZgJ|82Dsw;tP!{T^NFF~Z2yn?(K_2jdS%7tMc;CD|P; zR?N-s-K@}9Mn>{)M{GJzpKbZRq7o2yvH8x_u8hr`nv+OYf3xtLwy5i>E3gczyzBdl z9+G`yI}g|S6%0R#Jv5AzXY$2tXl6RtmDn@Z*pA-vx2sCY&5VAiRTYEm@SovXypFG5 zTp?s9Z=~wsq&xQklkZc+r)=tl?EUq%g07YBK<$SPV|lF9$0j9tLl5;8!ZWx-bg3dm zjnH5iQjp5lOQ%!OP9No6>E?}u!zvR1{?-HgVRjb98Gum1AB?9LNUABY6t9qy%yQt;b*TZP%p{7%9^GB z;mFm>UihKm0@n{PY-6uzWtj=?;`E^e^^9J~pWgKv{#SA9^7<~53*;@^+GHq+m| zT+Bc8J1nL3) zT?5jnd%3^%AVs-JI9`g<(9jmy>E~kh>EX^jI_{6zwWo)Afc=xuJlOfUQ}>Mrro_{_ z=Mk|1m`;nuf(?Ke!>fP)$^N^9_9PtH)$T4#)t)rKpK%z?Pc1Km1~ooK`}bb#68Xyz znJ@u2dR406(mm*WL)K-i=Dgp68q_Q?3H;81D0?O{{>EMd`M2U|N)bpzLai?REpUyq z#^o5nB_%H}dpQc3b#1rW8R0Z?dL;=BjCw$^hO=*SF7xI8= zqdDJuXDd=PO1!HpHtR)|-h+oG(}6OWT1f?E&5U~E4~r3`4t2I~RELUZd0+4=c6g%+HT`yZx{o!g`k zen&c8ndS;=fodUFq7G@F?3O@+g)7ANiN_nBeDnx#t5|hwWRoA;@Fzeafa{dZJ+_m2 zy(7W_?$r^ax}Gg&hkItSejUGXHSz6rTA-li6tAEM&KYzNt;t~Ry-6F>X!&LiLbkbN zEtNYRpVd0AFQ`Hml4R_`yl`4%GwlY}lHuu6g7p1x<(eYpHQ+E?PP$+5$H3b<1w*33 zvhtD9^fC+&c*_hn&o+W90PeqiY519xN4BH|ICS(?$_0Q;IyVfkMY4Lu?hT55^}rM} z3U}5Fm(FntnXR7JO=x%F{TU@lAqR~Qdzcz`-gq*1!0stKEUt9G(R6#j?gTEC61V9c zR{N6ZTI<>db2B?GaK#rCkhYXjROg*nd3wW>I^O<8+uq@0yRN|1c6pkm6P6Dz zcKa2XfCeOgIUo^nSBoy~HxcAMU!VCIC%~)(_Txuq;2vbQ>potN2WRS+M}`8|twhsdF03Eh=fq5-NJ>2% zsP4UN7T%A=l7->Rth3i_`Y~5?VMl_SpO|+QB^NDN1cdy#w{<>7)Mbl$5J9&)=>ey| zKNmW~JaRBNDQA1@(HGTE&B88HTe#CB7J=YFa8imXr)7glpyMd0tLpeecPSbD{xuN%Il(j{fvmf@GHLal!molTp8B1y_1{BBNQbdy4aLH? z8Kd+#wqB;Ts7R&|N;WaGw1>85cC|<9;MB##z8rP$!LGg`zo?+^hsAYy8AXLp-~dhM zSi-zJtGMQO#v34H^`fOBS3mEy^}1Amr+4_8#%;qw-Zccv6BP2yf)dAGHjCLljST4*S{f(9V+koIlqpm* zbWGi$i`q_(471w0*>a&Vx|ye-H=NPW5(6v)X}0`5Aq1jk*4ZHw8WU6!Qk zrSAs4V#$h3F)ImJM!=Pd+Cq)3uIXhvH%uE7x3E}`*>_#op^o99Kv|p$bhwVqGmE>q zb5+UmJ7pHxRX{^BMCd>AD_=!R53K6l)QsT(yGX{wJKz+4$ z@7euZc@^p6d+o>AzhZ&XK8auCe}-x2`Q3psX%2M%P#lF+Vj*~ztwv{tRAJi!xYA99 z=v2ix=TYM)qu?;|B_v)UB%aFv#oMfKJAXRTgmhs4(vhmZ1X&7q&S#`TIDl>#8yv<^ znA`w%?J?vdID2#Xa|Kb(rB5lD{X#E&A^!wK;_v8zn*cM}#(*%u9ALxJ1Jag7wb-L$ zvMigQ2ea38ZcxJcXKwnz74&;*96%}8k$4db1Sn`Y%*C&_8xhQee&<^zySck*kn6Q( z>cBLD)1&+8{jVVDF$e@tib77!MHH*o{8S_>#n#h6j#m~CD^oc6wWdAofoX^U^nTY- zN^x0LswJL<#JBG-ang&ap&8Of1_pt1R+wA-eEI}Z-Z2OM=@z&KFWhE8IsNM0Lb(){ zB7Q2=_6;xSo<5bFSzw5>!b{s|DWga&XWW?Bqbb?16q7f%F%9KX=p5 zy)USK?nxl*i+MuR!+&1V&(VS0UoKFCZcarN(er5U`SOOm9uEx=_T5Q;dg;G~eHYNd z!3LpbMDyp#=Wp&h0tc@vfgA?hjB}^yR~Rpa0$BDqtJGA;vj6jy)&e*a3%U8Q2LAnH kRP+!3sr>(<=a2c0rUnnzn_qiESJys!EGe8TsP*Q503~dQs{jB1 diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/2025-03-25_otel-delta-temporality-support.md index bc1cf6d..6c9649c 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/2025-03-25_otel-delta-temporality-support.md @@ -20,13 +20,13 @@ * [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q) * [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) -A proposal for adding experimental support for OTEL delta temporality metrics in Prometheus, allowing them be ingested, stored and queried directly. +A proposal for adding experimental support for OTEL delta temporality metrics in Prometheus, allowing them to be ingested, stored and queried directly. ## Why Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter-like OTEL metrics (e.g. histograms, sum) can have either [cumulative or delta temporality](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality). However, Prometheus only supports cumulative metrics, due to its pull-based approach to collecting metrics. -Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167[). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. +Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. The cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the [Pitfalls section](#pitfalls-of-the-current-solution) below. @@ -38,7 +38,7 @@ We propose some initial steps for delta support in this document. These delta fe In the [OTEL spec](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality), like cumulative metrics, a datapoint for a delta metric has a `(start,end]` time window. However, the time windows of delta datapoints do not overlap. -The `end` timestamp is called `TimeUnixNano` and is mandatory. The `start` timestamp is called `StartTimeUnixNano`. `StartTimeUnixNano` timestamp is optional, but recommended for better rate calculations and to detect gaps and overlaps in a stream. +The `end` timestamp is called `TimeUnixNano` and is mandatory. The `start` timestamp is called `StartTimeUnixNano`. `StartTimeUnixNano` timestamp is optional, but recommended for better rate calculations and to detect gaps and overlaps in a stream. ### Characteristics of delta metrics @@ -152,7 +152,7 @@ When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogra This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. -`--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on the that feature. +`--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on that feature. When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`), and the `__temporality__="delta"` label will be added. As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. @@ -164,9 +164,9 @@ Cumulative metrics ingested via the OTLP endpoint will also have a `__temporalit * When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) **Cons** -* Dependent the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usages for refinement. +* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usages for refinement. * Introduces additional complexity to the Prometheus data model. -* Systems or scripts that handle Prometheus metrics may be unware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. +* Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. * In this initial proposal, metrics with `__temporality__="delta"` will essentially be queried in the same way as Prometheus gauges - both gauges and deltas can be aggregated with `sum_over_time()`, so it may be confusing to have two different "types" with the same querying patterns. ### Metric names @@ -177,15 +177,15 @@ The `_total` suffix will not be added to OTEL deltas, ingested as either counter This means switching between cumulative and delta temporality can result in metric names changing, affecting dashboards and alerts. However, the current proposal requires different functions for querying delta and cumulative counters anyway. -### Monoticity +### Monotonicity -OTEL sums have a [monoticity property](https://opentelemetry.io/docs/specs/otel/metrics/supplementary-guidelines/#monotonicity-property), which indicates if the sum can only increase or if it can increase and decrease. Monotonic cumulative sums are mapped to Prometheus counters. Non-monotonic cumulative sums are mapped to Prometheus gauges, since Prometheus does not support counters that can decrease. This is because any drop in a Prometheus counter is assumed to be a counter reset. +OTEL sums have a [monotonicity property](https://opentelemetry.io/docs/specs/otel/metrics/supplementary-guidelines/#monotonicity-property), which indicates if the sum can only increase or if it can increase and decrease. Monotonic cumulative sums are mapped to Prometheus counters. Non-monotonic cumulative sums are mapped to Prometheus gauges, since Prometheus does not support counters that can decrease. This is because any drop in a Prometheus counter is assumed to be a counter reset. It is not necessary to detect counter resets for delta metrics - to get the increase over an interval, you can just sum the values over that interval. Therefore, for the `--enable-feature=otlp-native-delta-ingestion` option, where OTEL deltas are converted into Prometheus counters (with `__temporality__` label), non-monotonic delta sums will also be converted in the same way (with `__type__="counter"` and `__temporality__="delta"`). This ensures StatsD counters can be ingested as Prometheus counters. [The StatsD receiver sets counters as non monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). Note there has been some debate on whether this should be the case or not ([issue 1](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), [issue 2](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14956)). -Consequences include not being to convert delta counters in Prometheus into their cumulative counterparts (e.g. for any possible future querying extensions for deltas). Also, as monoticity information is lost, if the metrics are later exported back into the OTEL format, all deltas will have to be assumed to be non-monotonic. +Consequences include not being able to convert delta counters in Prometheus into their cumulative counterparts (e.g. for any possible future querying extensions for deltas). Also, as monoticity information is lost, if the metrics are later exported back into the OTEL format, all deltas will have to be assumed to be non-monotonic. ### Scraping @@ -209,7 +209,7 @@ Once deltas are ingested into Prometheus, they can be converted back into OTEL m The prometheusreceiver has the same issue described in [Scraping](#scraping) regarding possibly misaligned scrape vs delta ingestion intervals. -If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we gain more confience that the `__temporality__` label is the correct approach, the receiver should be updated to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. +If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we gain more confidence that the `__temporality__` label is the correct approach, the receiver should be updated to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. ### Querying deltas @@ -221,7 +221,7 @@ Having different functions for delta and cumulative counters mean that if the te Possible improvements to rate/increase calculations and user experience can be found in [Rate calculation extensions](#rate-calculation-extensions) and [Function overloading](#function-overloading). -Note: With [left-open range selectors](https://prometheus.io/docs/prometheus/3.5/migration/#range-selectors-and-lookback-exclude-samples-coinciding-with-the-left-boundary) introduced in Prometheus 3.0, queries such as `sum_over_time(metric[])` will exclude the sample at the left boundary. This is a fortunate usability improvement for querying deltas - with Prometheus 2, a `1m` interval actually covered `1m1s`, which could lead to double counting samples in consecutive steps and inflated sums; to get the actual value within `1m`, the awkward `59s999ms` had to be used instead. #### Querying range misalignment @@ -247,7 +247,7 @@ However, if you only query between T4 and T5, the rate would be 10/1 = 1 , and q Whether this is a problem or not is subjective. Users may prefer this behaviour, as unlike the cumulative `rate()`/`increase()`, it does not attempt to extrapolate. This makes the results easier to reason about and directly reflects the ingested data. The [Chronosphere user experience report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0) supports this: "user feedback indicated [`sum_over_time()`] felt much more natural and trustworthy when working with deltas" compared to converting deltas to cumulative and having `rate()`/`increase()` apply its usual extrapolation. -For some delta systems like StatsD, each sample represents an value that occurs at a specific moment in time, rather than being aggregated over a window. In these cases, each delta sample can be viewed as representing a infinitesimally small interval around its timestamp. This means taking into account of all the samples in the range, without extrapolation or interpolation, is an good representation of increase in the range - there are no samples in the range that only partially contribute to the range, and there are no samples out of the range which contribute to the increase in the range at all. For our initial implementation, the collection interval is ignored (i.e. `StartTimeUnixNano` is dropped), so all deltas could be viewed in this way. +For some delta systems like StatsD, each sample represents an value that occurs at a specific moment in time, rather than being aggregated over a window. In these cases, each delta sample can be viewed as representing a infinitesimally small interval around its timestamp. This means taking into account of all the samples in the range, without extrapolation or interpolation, is a good representation of increase in the range - there are no samples in the range that only partially contribute to the range, and there are no samples out of the range which contribute to the increase in the range at all. For our initial implementation, the collection interval is ignored (i.e. `StartTimeUnixNano` is dropped), so all deltas could be viewed in this way. #### Function warnings @@ -330,8 +330,8 @@ Downsides: * This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). * A possible adjustment is to just take the single value as the increase for the range. This may be more useful on average than returning no value in the case of a single sample. However, the mix of extrapolation and non-extrapolation logic may end up surprising users. If we do decide to generally extrapolate to fill the whole window, but have this special case for a single datapoint, someone might rely on the non-extrapolation behaviour and get surprised when there are two points and it changes. * Harder to predict the start and end of the series vs cumulative. -* The average spacing may not be a good estimation for the ingestion interval when delta metrics are sparse or or deliberately irregularly spaced (e.g. in the classic statsd use case). -* Additional downsides can be found in [this review comment](https://github.com/prometheus/proposals/pull/48#discussion_r2047990524)] for the proposal. +* The average spacing may not be a good estimation for the ingestion interval when delta metrics are sparse or deliberately irregularly spaced (e.g. in the classic statsd use case). +* Additional downsides can be found in [this review comment](https://github.com/prometheus/proposals/pull/48#discussion_r2047990524) for the proposal. Due to the numerous downsides, and the fact that more accurate lookahead/lookbehind techniques are already being explored for cumulative metrics (which means we could likely do something similar for deltas), it is unlikely that this option will actually be implemented. @@ -408,11 +408,11 @@ This also does not work for samples missing StartTimeUnixNano. #### Convert to rate on ingest -Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixName` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. +Convert delta metrics to per-second rate by dividing the sample value with (`TimeUnixNano` - `StartTimeUnixNano`) on ingest, and also append `:rate` to the end of the metric name (e.g. `http_server_request_duration_seconds` -> `http_server_request_duration_seconds:rate`). So the metric ends up looking like a normal Prometheus counter that was rated with a recording rule. The difference is that there is no interval information in the metric name (like :rate1m) as there is no guarantee that the interval from sample to sample stays constant. -To averages rates over more than the original collection interval, a new time-weighted average function is required to accommdate cases like the collection interval changing and having a query range which isn't a multiple of the interval. +To averages rates over more than the original collection interval, a new time-weighted average function is required to accommodate cases like the collection interval changing and having a query range which isn't a multiple of the interval. This would also require zero timestamp injection or CT-per-sample for better rate calculations. @@ -458,7 +458,7 @@ To work out the delta for all the cumulative native histograms in an range, the ### 1. Experimental feature flags for OTLP delta ingestion -Add the `--enable-feature=otlp-native-delta-ingestion` and `otlp-deltas-as-gauge` feature flags as described in [Delta metric type](#delta-metric-type) to add appropiate types and flags to series on ingestion. +Add the `--enable-feature=otlp-native-delta-ingestion` and `--enable-feature=otlp-delta-as-gauge-ingestion` feature flags as described in [Delta metric type](#delta-metric-type) to add appropriate types and flags to series on ingestion. Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any types to deltas. From 910c4de4a2a4436b9c90fa635e2c72ad0ba84838 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 18 Jul 2025 22:28:13 +0100 Subject: [PATCH 20/34] Renames Signed-off-by: Fiona Liao --- .../cumulative-smoothed.png | Bin ...=> 0048-otel_delta_temporality_support.md} | 64 +++++++++--------- 2 files changed, 33 insertions(+), 31 deletions(-) rename assets/{2025-03-25_otel-delta-temporality-support => 0048-otel_delta_temporality_support}/cumulative-smoothed.png (100%) rename proposals/{2025-03-25_otel-delta-temporality-support.md => 0048-otel_delta_temporality_support.md} (99%) diff --git a/assets/2025-03-25_otel-delta-temporality-support/cumulative-smoothed.png b/assets/0048-otel_delta_temporality_support/cumulative-smoothed.png similarity index 100% rename from assets/2025-03-25_otel-delta-temporality-support/cumulative-smoothed.png rename to assets/0048-otel_delta_temporality_support/cumulative-smoothed.png diff --git a/proposals/2025-03-25_otel-delta-temporality-support.md b/proposals/0048-otel_delta_temporality_support.md similarity index 99% rename from proposals/2025-03-25_otel-delta-temporality-support.md rename to proposals/0048-otel_delta_temporality_support.md index 6c9649c..e1205bf 100644 --- a/proposals/2025-03-25_otel-delta-temporality-support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -6,7 +6,7 @@ * **Contributors:** * Initial design started by @ArthurSens and @sh0rez - * Delta WG contributors: @ArthurSens, @enisoc and @subvocal + * Delta WG: @ArthurSens, @enisoc and @subvocal, among others * **Implementation Status:** `Partially implemented` @@ -102,6 +102,8 @@ These may come in later iterations of delta support. ## How +This section outlines the proposed approach for delta temporality support. Additional approaches and extensions can be found in [Potential future extensions](#potential-future-extensions) and [Discarded alternatives](#discarded-alternatives). + ### Ingesting deltas When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. @@ -261,6 +263,34 @@ There are also additional functions that should only be used with Prometheus gau This initial approach enables Prometheus to support OTEL delta metrics in a careful manner by relying on existing concepts like labels and reusing existing functions with minimal changes, and also provides a foundation for potentially building more advanced delta features in the future. +## Known unknowns + +### Native histograms performance + +To work out the delta for all the cumulative native histograms in an range, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the range, all of the buckets in all of the samples need to be summed, which is less efficient at query time. + +## Implementation Plan + +### 1. Experimental feature flags for OTLP delta ingestion + +Add the `--enable-feature=otlp-native-delta-ingestion` and `--enable-feature=otlp-delta-as-gauge-ingestion` feature flags as described in [Delta metric type](#delta-metric-type) to add appropriate types and flags to series on ingestion. + +Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any type metadata to the ingested series. + +### 2. Function warnings + +Add function warnings when a function is used with series of wrong type or temporality as described in [Function warnings](#function-warnings). + +There are already warnings if `rate()`/`increase()` are used without the `__type__="counter"` label: https://github.com/prometheus/prometheus/pull/16632. + +### 3. Update documentation + +Update documentation explaining new experimental delta functionality, including recommended configs for filtering out delta metrics from scraping and remote write. + +### 4. Review deltas in practice and experiment with possible future extensions + +Review how deltas work in practice using the current approach, and use experience and feedback to decide whether any of the potential extensions are necessary, and whether to discontinue one of the two options for delta ingestion (gauges vs. temporality label). + ## Potential future extensions Potential extensions, likely requiring dedicated proposals. @@ -303,7 +333,7 @@ The `smoothed` modifer in the extended range selectors proposal does this for cu The `smoothed` proposal works by injecting points at the edges of the range. For the start boundary, the injected point will have its value worked out by linearly interpolating between the closest point before the range start and the first point inside the range. -![cumulative smoothed example](../assets/2025-03-25_otel-delta-temporality-support/cumulative-smoothed.png) +![cumulative smoothed example](../assets/0048-otel_delta_temporality_support/cumulative-smoothed.png) That value would be nonesensical for deltas, as the values for delta samples are independent. Additionally, for deltas, to work out the increase, we add all the values up in the range (with some adjustments) vs in the cumulative case where you subtract the first point in the range from the last point. So it makes sense the smoothing behaviour would be different. @@ -446,32 +476,4 @@ Delta to cumulative conversion at query time doesn’t have the same out of orde No function modification is needed - all cumulative functions will work for samples ingested as deltas. -However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. - -## Known unknowns - -### Native histograms performance - -To work out the delta for all the cumulative native histograms in an range, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the range, all of the buckets in all of the samples need to be summed, which is less efficient at query time. - -## Implementation Plan - -### 1. Experimental feature flags for OTLP delta ingestion - -Add the `--enable-feature=otlp-native-delta-ingestion` and `--enable-feature=otlp-delta-as-gauge-ingestion` feature flags as described in [Delta metric type](#delta-metric-type) to add appropriate types and flags to series on ingestion. - -Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any types to deltas. - -### 2. Function warnings - -Add function warnings when a function is used with series of wrong type or temporality as described in [Function warnings](#function-warnings). - -There are already warnings if `rate()`/`increase()` are used without the `__type__="counter"` label: https://github.com/prometheus/prometheus/pull/16632. - -### 3. Update documentation - -Update documentation explaining new experimental delta functionality, including recommended configs for filtering out delta metrics from scraping and remote write. - -### 4. Review deltas in practice and experiment with possible future extensions - -Review how deltas work in practice using the current approach, and use experience and feedback to decide whether any of the potential extensions are necessary, and whether to discontinue one of the two options for delta ingestion (gauges vs. temporality label). +However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. \ No newline at end of file From 6b2bc91177a8914dad0996443ff2ec3629c51076 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 18 Jul 2025 23:50:56 +0100 Subject: [PATCH 21/34] Small edits for clarity Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 44 ++++++++++++------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index e1205bf..f8bf560 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -4,7 +4,7 @@ * **Owners:** * @fionaliao -* **Contributors:** +* **Contributors:** * Initial design started by @ArthurSens and @sh0rez * Delta WG: @ArthurSens, @enisoc and @subvocal, among others @@ -42,15 +42,19 @@ The `end` timestamp is called `TimeUnixNano` and is mandatory. The `start` times ### Characteristics of delta metrics -Sparse metrics are more common for delta than cumulative metrics. While delta datapoints can be emitted at a regular interval, in some cases (like the OTEL SDKs), datapoints are only emitted when there is a change (e.g. if tracking request count, only send a datapoint if the number of requests in the ingestion interval > 0). This can be beneficial for the metrics producer, reducing memory usage and network bandwidth. +### Sparseness -Further insights and discussions on delta metrics can be found in [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y), which describes Chronosphere's experience of adding functionality to ingest OTEL delta metrics and query them back with PromQL, and also [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q). +Sparse metrics are more common for delta than cumulative metrics. While delta datapoints can be emitted at a regular interval, in some cases (like the OTEL SDKs), datapoints are only emitted when there is a change (e.g. if tracking request count, only send a datapoint if the number of requests in the collection interval > 0). This can be beneficial for the metrics producer, reducing memory usage and network bandwidth. #### Alignment -The Prometheus scrape collection model deliberately gives you "unaligned" sampling, i.e. targets with the same scrape interval are still scraped at different phases (not all at the full minute, but hashed over the minute). +Usually, delta metrics are reported at timestamps aligned to the collection interval - that is, values are collected over a defined window (for example, one minute) and the final aggregated value is emitted exactly at the interval boundary (such as at each full minute). -The usual case for delta metrics is to collect increments over the collection interval (e.g. 1m), and then send out the collected increments at the full minute. This isn't true in all cases though, for example, the StatsD client libraries emits a delta every time an event happens rather than aggregating, producing unaligned samples (though the StatsD daemon does then aggregate to an aligned interval). +This isn't true in all cases though, for example, the StatsD client libraries emits a delta every time an event happens rather than aggregating, producing unaligned samples (though if the StatsD daemon is used, that then aggregates the input values and aligns the data to the interval). + +#### Further reading + +[Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) describes Chronosphere's experience of adding functionality to ingest OTEL delta metrics and query them back with PromQL, and there is additionally [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q). ### Pitfalls of the current solution @@ -120,7 +124,7 @@ Delta histograms will use native histogram chunks with the `GaugeType` counter r ### Delta metric type -It is useful to be able to distinguish between delta and cumulative metrics. This would allow users to understand what the raw data represents and what functions are appropiate to use. Additionally, this could allow the query engine or UIs displaying Prometheus data apply different behaviour depending on the metric type to provide meaningful output. +It is useful to be able to distinguish between delta and cumulative metrics. This would allow users to understand what the raw data represents and what functions are appropriate to use. Additionally, this could allow the query engine or UIs displaying Prometheus data apply different behaviour depending on the metric type to provide meaningful output. As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. @@ -142,12 +146,12 @@ When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogra **Pros** * Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. -* Prometheus already implicitly uses gauges to represent deltas. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. +* Prometheus already uses gauges to represent deltas in some cases. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. * Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. **Cons** * Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. -* Gauges are usually aggregated in time by averaging or taking the last value, while deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics (e.g. the [Grafana Explore Metrics](https://github.com/grafana/grafana/blob/main/docs/sources/explore/_index.md) feature). +* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics (e.g. the [Grafana Explore Metrics](https://github.com/grafana/grafana/blob/main/docs/sources/explore/_index.md) feature). * The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. #### Introduce `__temporality__` label @@ -166,10 +170,10 @@ Cumulative metrics ingested via the OTLP endpoint will also have a `__temporalit * When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) **Cons** -* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usages for refinement. +* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Introduces additional complexity to the Prometheus data model. * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. -* In this initial proposal, metrics with `__temporality__="delta"` will essentially be queried in the same way as Prometheus gauges - both gauges and deltas can be aggregated with `sum_over_time()`, so it may be confusing to have two different "types" with the same querying patterns. +* For the initial proposal, the difference between metrics with `__temporality__="delta"` and gauges is mainly just informational. `sum_over_time()` can be used for both (though it might not be appropiate for all gauge metrics). It can be confusing that the same function can be used for two distinct "types". ### Metric names @@ -195,7 +199,11 @@ No scraped metrics should have delta temporality as there is no additional benef ### Federation -Federating delta series directly could be usable if there is a constant and known collection interval for the delta series, and the metrics are scraped at least as often as the collection interval. This is not the case for all deltas and the scrape interval cannot be enforced. Therefore we will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the `__temporality__="delta"` label is set. If deltas are converted to gauges, there would not be a way to distinguish deltas from regular gauges so we cannot provide a scrape config. +Federating delta series directly could be usable if there is a constant and known collection interval for the delta series, and the metrics are scraped at least as often as the collection interval. However, this is not the case for all deltas and the scrape interval cannot be enforced. + +Therefore we will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the `__temporality__="delta"` label is set. + +If deltas are converted to gauges, there would not be a way to distinguish deltas from regular gauges so we cannot provide a scrape config. ### Remote write ingestion @@ -207,9 +215,9 @@ Prometheus has metric metadata as part of its metric model, which include the ty ### Prometheus OTEL receivers -Once deltas are ingested into Prometheus, they can be converted back into OTEL metrics by the prometheusreceiver (scrape) and [prometheusremotewritereceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusremotewritereceiver) (push). +Once deltas are ingested into Prometheus, they can be converted back into OTEL metrics by the [prometheusreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver) (scrape) and [prometheusremotewritereceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusremotewritereceiver) (push). -The prometheusreceiver has the same issue described in [Scraping](#scraping) regarding possibly misaligned scrape vs delta ingestion intervals. +The prometheusreceiver has the same issue described in [Scraping](#scraping) regarding possibly misaligned scrape vs delta collection intervals. If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we gain more confidence that the `__temporality__` label is the correct approach, the receiver should be updated to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. @@ -217,7 +225,7 @@ If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-n For this initial proposal, existing functions will be used for querying deltas. -`rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[]) / ` can be used for the rate. `metric / interval` can also be used to calculate a rate if the ingestion interval is known. +`rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[]) / ` can be used for the rate. `metric / interval` can also be used to calculate a rate if the collection interval is known. Having different functions for delta and cumulative counters mean that if the temporality of a metric changes, queries will have to be updated. @@ -251,6 +259,8 @@ Whether this is a problem or not is subjective. Users may prefer this behaviour, For some delta systems like StatsD, each sample represents an value that occurs at a specific moment in time, rather than being aggregated over a window. In these cases, each delta sample can be viewed as representing a infinitesimally small interval around its timestamp. This means taking into account of all the samples in the range, without extrapolation or interpolation, is a good representation of increase in the range - there are no samples in the range that only partially contribute to the range, and there are no samples out of the range which contribute to the increase in the range at all. For our initial implementation, the collection interval is ignored (i.e. `StartTimeUnixNano` is dropped), so all deltas could be viewed in this way. +Additionally, as mentioned before, it is comment for deltas to have samples emitted at fixed time boundaries (i.e. are aligned). This means if the collection interval is known, and query ranges match the time boundaries, accurate results can be produces with `sum_over_time()`. + #### Function warnings To help users use the correct functions, warnings will be added if the metric type/temporality does not match the types that should be used with the function. @@ -297,13 +307,13 @@ Potential extensions, likely requiring dedicated proposals. ### CT-per-sample -[CreatedTimestamp (PROM-29)](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md) records when a time series was created or last reset, therefore allowing more accurate rate or increase calculations. This is similar to the `StartTimeUnixNano` field in OTEL metrics. +[CreatedTimestamp (PROM-29)](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md) records when a series was created or last reset, therefore allowing more accurate rate or increase calculations. This is similar to the `StartTimeUnixNano` field in OTEL metrics. There is an effort towards adding CreatedTimestamp as a field for each sample ([PR](https://github.com/prometheus/prometheus/pull/16046/files)). This is for cumulative counters, but can be reused for deltas too. When this is completed, if `StartTimeUnixNano` is set for a delta counter, it should be stored in the CreatedTimestamp field of the sample. CT-per-sample is not a blocker for deltas - before this is ready, `StartTimeUnixNano` will just be ignored. -Having CT-per-sample can improve the `rate()` calculation - the ingestion interval for each sample will be directly available, rather than having to guess the interval based on gaps. It also means a single sample in the range can result in a result from `rate()` as the range will effectively have an additional point at `StartTimeUnixNano`. +Having CT-per-sample can improve the `rate()` calculation - the collection interval for each sample will be directly available, rather than having to guess the interval based on gaps. It also means a single sample in the range can result in a result from `rate()` as the range will effectively have an additional point at `StartTimeUnixNano`. There are unknowns over the performance and storage of essentially doubling the number of samples with this approach. @@ -360,7 +370,7 @@ Downsides: * This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). * A possible adjustment is to just take the single value as the increase for the range. This may be more useful on average than returning no value in the case of a single sample. However, the mix of extrapolation and non-extrapolation logic may end up surprising users. If we do decide to generally extrapolate to fill the whole window, but have this special case for a single datapoint, someone might rely on the non-extrapolation behaviour and get surprised when there are two points and it changes. * Harder to predict the start and end of the series vs cumulative. -* The average spacing may not be a good estimation for the ingestion interval when delta metrics are sparse or deliberately irregularly spaced (e.g. in the classic statsd use case). +* The average spacing may not be a good estimation for the collection interval when delta metrics are sparse or deliberately irregularly spaced (e.g. in the classic statsd use case). * Additional downsides can be found in [this review comment](https://github.com/prometheus/proposals/pull/48#discussion_r2047990524) for the proposal. Due to the numerous downsides, and the fact that more accurate lookahead/lookbehind techniques are already being explored for cumulative metrics (which means we could likely do something similar for deltas), it is unlikely that this option will actually be implemented. From 06d9a52f2a02de26d53587a1c677f85dc698d667 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Mon, 21 Jul 2025 09:31:13 +0100 Subject: [PATCH 22/34] Add section for otel properties as labels Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 64 +++++++++++++------ 1 file changed, 43 insertions(+), 21 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index f8bf560..b7ebd05 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -134,26 +134,30 @@ We propose to add two options as feature flags for ingesting deltas: 2. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series. -We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta-like use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but it has the potential to offer a better user experience in the long run as it allows more precise differentiation. +We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but is a closer fit to the OTEL model. Below we explore the pros and cons of each option in more detail. #### Treat as gauge -Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. +Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. + +Gauges ingested into Prometheus via scraping represent sampled values for the metric and using `sum_over_time()` for these types of gauges do not make sense. However, there are other sources that can ingest "delta" gauges already for which summing does make sense. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. **Pros** * Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. -* Prometheus already uses gauges to represent deltas in some cases. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. +* Prometheus already uses gauges to represent deltas. For example, `increase()` outputs the delta of a counter series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. * Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. **Cons** * Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. -* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics (e.g. the [Grafana Explore Metrics](https://github.com/grafana/grafana/blob/main/docs/sources/explore/_index.md) feature). +* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics. * The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. +The cons are generally around it being difficult to tell apart OTEL gauges and counters when ingested into Prometheus. An extension could be to [Add otel metric properties as labels](#add-otel-metric-properties-as-labels), so there is extra information users can use to decide on how to query the metric, while the Prometheus type remains a gauge. + #### Introduce `__temporality__` label This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. @@ -165,15 +169,16 @@ When ingesting a delta metric via the OTLP endpoint, the metric type is set to ` Cumulative metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. **Pros** -* Clear distinction between delta metrics and gauge metrics. +* Clear distinction between OTEL delta metrics and OTEL gauge metrics when ingested into Prometheus, meaning users know the appriopiate functions to use for each type (e.g. `sum_over_time()` is unlikely to be useful for OTEL gauge metrics). * Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having separate `__type__` and `__temporality__` labels in Prometheus. * When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) **Cons** -* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Introduces additional complexity to the Prometheus data model. +* Confusing overlap between gauge and `__temporality__="delta"`. As mentioned in [Treat as gauge](#treat-as-gauge), essentially deltas already exist in Prometheus as gauges, and deltas can be viewed as a subset of gauges under the Prometheus definition. The same `sum_over_time()` would be used for aggregating these pre-existing deltas-as-gauges and OTEL deltas with counter type and `__temporality__="delta"`, creating confusion on why there are two different "types". + * Pre-existing deltas-as-gauges to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time. +* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. -* For the initial proposal, the difference between metrics with `__temporality__="delta"` and gauges is mainly just informational. `sum_over_time()` can be used for both (though it might not be appropiate for all gauge metrics). It can be confusing that the same function can be used for two distinct "types". ### Metric names @@ -257,9 +262,9 @@ However, if you only query between T4 and T5, the rate would be 10/1 = 1 , and q Whether this is a problem or not is subjective. Users may prefer this behaviour, as unlike the cumulative `rate()`/`increase()`, it does not attempt to extrapolate. This makes the results easier to reason about and directly reflects the ingested data. The [Chronosphere user experience report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0) supports this: "user feedback indicated [`sum_over_time()`] felt much more natural and trustworthy when working with deltas" compared to converting deltas to cumulative and having `rate()`/`increase()` apply its usual extrapolation. -For some delta systems like StatsD, each sample represents an value that occurs at a specific moment in time, rather than being aggregated over a window. In these cases, each delta sample can be viewed as representing a infinitesimally small interval around its timestamp. This means taking into account of all the samples in the range, without extrapolation or interpolation, is a good representation of increase in the range - there are no samples in the range that only partially contribute to the range, and there are no samples out of the range which contribute to the increase in the range at all. For our initial implementation, the collection interval is ignored (i.e. `StartTimeUnixNano` is dropped), so all deltas could be viewed in this way. +For some delta systems like StatsD, each sample represents an value that occurs at a specific moment in time, rather than being aggregated over a window. In these cases, each delta sample can be viewed as representing an infinitesimally small interval around its timestamp. This means taking into account of all the samples in the range, without extrapolation or interpolation, is a good representation of increase in the range - there are no samples in the range that only partially contribute to the range, and there are no samples out of the range which contribute to the increase in the range at all. For our initial implementation, the collection interval is ignored (i.e. `StartTimeUnixNano` is dropped), so all deltas could be viewed in this way. -Additionally, as mentioned before, it is comment for deltas to have samples emitted at fixed time boundaries (i.e. are aligned). This means if the collection interval is known, and query ranges match the time boundaries, accurate results can be produces with `sum_over_time()`. +Additionally, as mentioned before, it is common for deltas to have samples emitted at fixed time boundaries (i.e. are aligned). This means if the collection interval is known, and query ranges match the time boundaries, accurate results can be produces with `sum_over_time()`. #### Function warnings @@ -267,7 +272,6 @@ To help users use the correct functions, warnings will be added if the metric ty * Cumulative counter-specific functions (`rate()`, `increase()`, `irate()` and `resets()`) will warn if `__type__="gauge"` or `__temporality__="delta"`. * `sum_over_time()` will warn if `type="counter"` with no `__temporality__` label (implies cumulative counter), or if `__temporality__="cumulative"`. -There are also additional functions that should only be used with Prometheus gauges (e.g. `delta()`) rather than cumulative counters. Out of scope of delta vs cumulative though. ### Summary @@ -303,7 +307,7 @@ Review how deltas work in practice using the current approach, and use experienc ## Potential future extensions -Potential extensions, likely requiring dedicated proposals. +Potential extensions, some may require dedicated proposals. ### CT-per-sample @@ -347,7 +351,9 @@ The `smoothed` proposal works by injecting points at the edges of the range. For That value would be nonesensical for deltas, as the values for delta samples are independent. Additionally, for deltas, to work out the increase, we add all the values up in the range (with some adjustments) vs in the cumulative case where you subtract the first point in the range from the last point. So it makes sense the smoothing behaviour would be different. -In the delta case, the adjustment to samples in the range used for the rate calculation would be to work out the proportion of the first sample within the range and update its value. We would use the assumption that the start timestamp for the first sample is equal the the timestamp of the previous sample, and then use the formula `inside value * (inside ts - range start ts) / (inside ts - outside ts)` to adjust the first sample (aka the `inside value`). +In the delta case, the adjustment to samples in the range used for the rate calculation would be to work out the proportion of the first sample within the range and update its value. We would use the assumption that the start timestamp for the first sample is equal the the timestamp of the previous sample, and then use the formula `inside value * (inside ts - range start ts) / (inside ts - outside ts)` to adjust the first sample (aka the `inside value`). + +Support for retaining `StartTimeUnixNano` would improve the calculation, as we would then know the real interval for the first sample, rather than assuming its interval extends back until the previous sample. For sparse cases, this assumption could be wrong. #### Similar logic to cumulative case @@ -355,7 +361,7 @@ For cumulative counters, `increase()` works by subtracting the first sample from For consistency, we could emulate that for deltas. -First sum all sample values in the range, with the first sample’s value only partially included if it's not completely within the query range. To estimate the proporation of the first sample within the range, assume its interval is the average interval betweens all samples within the range. If the last sample does not align with the end of the time range, the sum should be extrapolated to cover the range until the end boundary. +First sum all sample values in the range, with the first sample’s value only partially included if it's not completely within the query range. To estimate the proportion of the first sample within the range, assume its interval is the average interval betweens all samples within the range. If the last sample does not align with the end of the time range, the sum should be extrapolated to cover the range until the end boundary. The cumulative `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. @@ -370,15 +376,17 @@ Downsides: * This will not work if there is only a single sample in the range, which is more likely with delta metrics (due to sparseness, or being used in short-lived jobs). * A possible adjustment is to just take the single value as the increase for the range. This may be more useful on average than returning no value in the case of a single sample. However, the mix of extrapolation and non-extrapolation logic may end up surprising users. If we do decide to generally extrapolate to fill the whole window, but have this special case for a single datapoint, someone might rely on the non-extrapolation behaviour and get surprised when there are two points and it changes. * Harder to predict the start and end of the series vs cumulative. -* The average spacing may not be a good estimation for the collection interval when delta metrics are sparse or deliberately irregularly spaced (e.g. in the classic statsd use case). -* Additional downsides can be found in [this review comment](https://github.com/prometheus/proposals/pull/48#discussion_r2047990524) for the proposal. +* The average spacing may not be a good estimation for the collection interval when delta metrics are sparse or irregularly spaced. +* Additional downsides can be found in [this review comment](https://github.com/prometheus/proposals/pull/48#discussion_r2047990524). -Due to the numerous downsides, and the fact that more accurate lookahead/lookbehind techniques are already being explored for cumulative metrics (which means we could likely do something similar for deltas), it is unlikely that this option will actually be implemented. +The numerous downsides likely outweigh the argument for consistency. Additionally, lookahead/lookbehind techniques are already being explored for cumulative metrics with the `smoothed` modifier. This means we could likely do something similar for deltas. Lookahead/lookbehind aims to do a similar thing to this calculation - estimate the increase/rate for the series within the selected query range. However, since it can look beyond the boundary of the query range, it should get better results. ### Function overloading `rate()` and `increase()` could be extended to work transparently with both cumulative and delta metrics. The PromQL engine could check the `__temporality__` label and execute the correct logic. +Function overloading could also work if OTEL deltas are ingested as Prometheus gauges and the `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on gauges that are unsuited to being summed over time, not add any warnings, and produce nonsensical results. + Pros: * Users would not need to know the temporality of their metric to write queries. Users often don’t know or may not be able to control the temporality of a metric (e.g. if they instrument the application, but the metric processing pipeline run by another team changes the temporality). @@ -388,6 +396,8 @@ Pros: Cons: * The increased internal complexity could end up being more confusing. +* Inconsistent functions depending on type and temporality. Both `increase()` and `sum_over_time()` could be used for aggregating deltas over time. However, `sum_over_time()` would not work for cumulative metrics, and `increase()` would not work for gauges. + * This gets further complicated when considering how functions could be combined. For example, a user may use recording rules to compute the sum of a delta metric over a shorter period of time, so they can later for the sum over a longer period of time faster by using the pre-computed results. `increase()` and `sum_over_time()` would both work for the recording rule. For the long query, `increase()` will not work - it would return a warning (as the pre-computed values would be gauges/untyped and should not be used with `increase()`) and a wrong value (if the cumulative counter logic is applied when there is no `__temporality__="delta"` label). `sum_over_time()` for the long query would work in the expected manner, however. * Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). * One possible extension could be to stitch the cumulative and delta series together and return a single result. * There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. @@ -398,22 +408,18 @@ Open questions and considerations: * There are open questions on how to best calculate the rate or increase of delta metrics (see [`rate()` behaviour for deltas](#rate-behaviour-for-deltas) below), and there is currently ongoing work with [extending range selectors for cumulative counters](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md), which should be taken into account for deltas too. * Once we start with overloading functions, users may ask for more of that e.g. should we change `sum_over_time()` to also allow calculating the increase of cumulative metrics rather than just summing samples together. Where would the line be in terms of which functions should be overloaded or not? One option would be to only allow `rate()` and `increase()` to be overloaded, as they are the most popular functions that would be used with counters. -Function overloading could also technically work if OTEL deltas are ingested as Prometheus gauges and the `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on actual gauges (e.g. max cpu), not add any warnings, and produce nonsensical results. - #### `rate()` behaviour for deltas If we were to implement function overloading for `rate()` and `increase()`, how exactly will it behave for deltas? A few possible ways to do rate calculation have been outlined, each with their own pros and cons. Also to take into account are the new `smoothed` and `anchored` modifiers in the extended range selectors proposal. -The current proposed solution would be: +A possible proposal would be: * no modifier - just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). * `anchored` - same as no modifer. In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas. To get the same output in the cumulative and delta cases given the same input to the initial instrumented counter, the delta case should use `sum_over_time()`. * `smoothed` - Logic as described in [Lookahead and lookbehind](#lookahead-and-lookbehind-of-range). -For the no modifier case, the most consistent behaviour with to cumulative metrics would be do implement what's describe in Similar logic to cumulative case. This could result in fewer surprises if switching between delta and cumulative. However, the extrapolating behaviour does not work well for deltas in many cases, so it's unlikely we will go down that route. - One problem with reusing the range selector modifiers is that they are more generic than just modifiers for `rate()` and `increase()`, so adding delta-specific logic for these modifiers for `rate()` and `increase()` may be confusing. #### How to proceed @@ -432,6 +438,22 @@ A possible future enhancement is to add an `__monotonicity__` label along with ` Additionally, if there were a reliable way to have [Created Timestamp](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md) for all cumulative counters, we could consider supporting non-monotonic cumulative counters as well, as at that point the created timestamp could be used for working out counter resets instead of decreases in counter value. This may not be feasible or wanted in all cases though. +### Add otel metric properties as labels + +This would primarily be beneficial for the `--enable-feature=otlp-delta-as-gauge-ingestion` option. + +The key problem of ingesting OTEL deltas as Prometheus gauges is the lack of distinction between OTEL gauges and OTEL deltas, even though their semantics differ. Adding the original OTEL types as labels means that users would be able to get this information. For an OTEL delta sum, its labels would include `__type__="gauge"`, `__otel_type__="sum"` and `__otel_temporality__="delta"` (and `__otel_monotonicity__="true"/"false"`). This could also be useful for non-delta use cases, for example, being able to identify OTEL non-monotonic cumulative counters vs OTEL gauges. + +Function overloading could still be done - `rate()` and `increase()` could work based on the OTEL type label, rather than a general temporality label. + +This approach makes delta support OTEL-specific, rather than a more generic Prometheus feature. However, Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. + +There is a risk that tooling and workflows could rely too much on the OTEL-specific labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. + +This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also a separate type label to consider. + +An additional consideration would be any CreatedTimestamp features would need to work for both Prometheus counters and gauges, so that `StartTimeUnixNano` would be able to be preserved for deltas-as-gauges. + ## Discarded alternatives ### Ingesting deltas alternatives From 7dd78c32d0cfce77e7ee7d67e04e37fba2290990 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Mon, 21 Jul 2025 10:20:56 +0100 Subject: [PATCH 23/34] fix sparseness header Signed-off-by: Fiona Liao --- proposals/0048-otel_delta_temporality_support.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index b7ebd05..802f8e7 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -42,7 +42,7 @@ The `end` timestamp is called `TimeUnixNano` and is mandatory. The `start` times ### Characteristics of delta metrics -### Sparseness +#### Sparseness Sparse metrics are more common for delta than cumulative metrics. While delta datapoints can be emitted at a regular interval, in some cases (like the OTEL SDKs), datapoints are only emitted when there is a change (e.g. if tracking request count, only send a datapoint if the number of requests in the collection interval > 0). This can be beneficial for the metrics producer, reducing memory usage and network bandwidth. From 46722dc90bdfdd339e3fb058111a110dcd8c3a7e Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 22 Jul 2025 11:43:17 +0100 Subject: [PATCH 24/34] Move preferred approach (__temporality__) to top Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 61 ++++++++++--------- 1 file changed, 32 insertions(+), 29 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index 802f8e7..81da507 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -130,34 +130,14 @@ As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_typ We propose to add two options as feature flags for ingesting deltas: -1. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges. +1. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series. -2. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series. +2. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges. -We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but is a closer fit to the OTEL model. +We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but it brings Prometheus more in alignment with the OTEL model. The preferred approach from the Prometheus delta working group is the `__temporality__` option, but we need practical experience to validate this is actually the case. Below we explore the pros and cons of each option in more detail. -#### Treat as gauge - -Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. - -Gauges ingested into Prometheus via scraping represent sampled values for the metric and using `sum_over_time()` for these types of gauges do not make sense. However, there are other sources that can ingest "delta" gauges already for which summing does make sense. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. - -When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. - -**Pros** -* Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. -* Prometheus already uses gauges to represent deltas. For example, `increase()` outputs the delta of a counter series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. -* Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. - -**Cons** -* Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. -* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics. -* The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. - -The cons are generally around it being difficult to tell apart OTEL gauges and counters when ingested into Prometheus. An extension could be to [Add otel metric properties as labels](#add-otel-metric-properties-as-labels), so there is extra information users can use to decide on how to query the metric, while the Prometheus type remains a gauge. - #### Introduce `__temporality__` label This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. @@ -176,10 +156,31 @@ Cumulative metrics ingested via the OTLP endpoint will also have a `__temporalit **Cons** * Introduces additional complexity to the Prometheus data model. * Confusing overlap between gauge and `__temporality__="delta"`. As mentioned in [Treat as gauge](#treat-as-gauge), essentially deltas already exist in Prometheus as gauges, and deltas can be viewed as a subset of gauges under the Prometheus definition. The same `sum_over_time()` would be used for aggregating these pre-existing deltas-as-gauges and OTEL deltas with counter type and `__temporality__="delta"`, creating confusion on why there are two different "types". - * Pre-existing deltas-as-gauges to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time. + * Pre-existing deltas-as-gauges could be converted to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time. * Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. + +#### Treat as gauge + +Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. + +Gauges ingested into Prometheus via scraping represent sampled values for the metric and using `sum_over_time()` for these types of gauges do not make sense. However, there are other sources that can ingest "delta" gauges already for which summing does make sense. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. + +When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. + +**Pros** +* Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. +* Prometheus already uses gauges to represent deltas. For example, `increase()` outputs the delta of a counter series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. +* Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. + +**Cons** +* Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. +* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics. +* The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. + +The cons are generally around it being difficult to tell apart OTEL gauges and counters when ingested into Prometheus. An extension could be to [Add otel metric properties as labels](#add-otel-metric-properties-as-labels), so there is extra information users can use to decide on how to query the metric, while the Prometheus type remains a gauge. + ### Metric names OTEL metric names are normalised when translated to Prometheus by default ([code](https://github.com/prometheus/otlptranslator/blob/94f535e0c5880f8902ab8c7f13e572cfdcf2f18e/metric_namer.go#L157)). This includes adding suffixes in some cases. For example, OTEL metrics converted into Prometheus counters (i.e. monotonic cumulative sums in OTEL) have the `__total` suffix added to the metric name, while gauges do not. @@ -228,7 +229,7 @@ If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-n ### Querying deltas -For this initial proposal, existing functions will be used for querying deltas. +For this initial proposal, existing functions will be used for querying deltas. This works for both the `__temporality__` and delta-as-gauge options. `rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[]) / ` can be used for the rate. `metric / interval` can also be used to calculate a rate if the collection interval is known. @@ -482,10 +483,6 @@ Users might want to convert back to original values (e.g. to sum the original va This also does not work for samples missing StartTimeUnixNano. -#### Map non-monotonic delta counters to gauges - -Mapping non-monotonic delta counters to gauges would be problematic, as it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Different functions would be needed for non-monotonic counters of differerent temporalities. - ### Delta metric type alternatives #### Add delta `__type__` label values @@ -500,6 +497,12 @@ Additionally, combining temporality and type means that every time a new type is Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. +### Monotonicity alternatives + +#### Map non-monotonic delta counters to gauges with `__temporality__` option + +With the `__temporality__` option, we could map monotonic deltas to the counter type, and non-monotonic counters to gauges. However, it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name), without adding [additional otel metric properties as labels](#add-otel-metric-properties-as-labels). + ### Querying deltas alternatives #### Convert to cumulative on query From d5b074acacdfdc5765e5f4caa1ee8e04394b9106 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 22 Jul 2025 11:54:46 +0100 Subject: [PATCH 25/34] Explain why statsd is important Signed-off-by: Fiona Liao --- proposals/0048-otel_delta_temporality_support.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index 81da507..a625644 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -160,7 +160,6 @@ Cumulative metrics ingested via the OTLP endpoint will also have a `__temporalit * Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. - #### Treat as gauge Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. @@ -195,7 +194,7 @@ OTEL sums have a [monotonicity property](https://opentelemetry.io/docs/specs/ote It is not necessary to detect counter resets for delta metrics - to get the increase over an interval, you can just sum the values over that interval. Therefore, for the `--enable-feature=otlp-native-delta-ingestion` option, where OTEL deltas are converted into Prometheus counters (with `__temporality__` label), non-monotonic delta sums will also be converted in the same way (with `__type__="counter"` and `__temporality__="delta"`). -This ensures StatsD counters can be ingested as Prometheus counters. [The StatsD receiver sets counters as non monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). Note there has been some debate on whether this should be the case or not ([issue 1](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), [issue 2](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14956)). +This ensures StatsD counters can be ingested as Prometheus counters. Since StatsD is so widely used, it's important to make sure StatsD counters (ingested via OTEL) will work properly in Prometheus. [StatsD counters are non-monotonic by definition](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), and [the StatsD receiver sets counters as non-monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). Consequences include not being able to convert delta counters in Prometheus into their cumulative counterparts (e.g. for any possible future querying extensions for deltas). Also, as monoticity information is lost, if the metrics are later exported back into the OTEL format, all deltas will have to be assumed to be non-monotonic. @@ -263,9 +262,9 @@ However, if you only query between T4 and T5, the rate would be 10/1 = 1 , and q Whether this is a problem or not is subjective. Users may prefer this behaviour, as unlike the cumulative `rate()`/`increase()`, it does not attempt to extrapolate. This makes the results easier to reason about and directly reflects the ingested data. The [Chronosphere user experience report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0) supports this: "user feedback indicated [`sum_over_time()`] felt much more natural and trustworthy when working with deltas" compared to converting deltas to cumulative and having `rate()`/`increase()` apply its usual extrapolation. -For some delta systems like StatsD, each sample represents an value that occurs at a specific moment in time, rather than being aggregated over a window. In these cases, each delta sample can be viewed as representing an infinitesimally small interval around its timestamp. This means taking into account of all the samples in the range, without extrapolation or interpolation, is a good representation of increase in the range - there are no samples in the range that only partially contribute to the range, and there are no samples out of the range which contribute to the increase in the range at all. For our initial implementation, the collection interval is ignored (i.e. `StartTimeUnixNano` is dropped), so all deltas could be viewed in this way. +For some delta systems like StatsD, each sample represents an value that occurs at a specific moment in time, rather than being aggregated over a window. Each delta sample can be viewed as representing an infinitesimally small interval around its timestamp. This means taking into account of all the samples in the range, without extrapolation or interpolation, is a good representation of increase in the range - there are no samples in the range that only partially contribute to the range, and there are no samples out of the range which contribute to the increase in the range at all. For our initial implementation, the collection interval is ignored (i.e. `StartTimeUnixNano` is dropped), so all deltas could be viewed in this way. -Additionally, as mentioned before, it is common for deltas to have samples emitted at fixed time boundaries (i.e. are aligned). This means if the collection interval is known, and query ranges match the time boundaries, accurate results can be produces with `sum_over_time()`. +Additionally, as mentioned before, it is common for deltas to have samples emitted at fixed time boundaries (i.e. are aligned). This means if the collection interval is known, and query ranges match the time boundaries, accurate results can be produced with `sum_over_time()`. #### Function warnings @@ -501,7 +500,7 @@ Have a convention for naming metrics e.g. appending `_delta_counter` to a metric #### Map non-monotonic delta counters to gauges with `__temporality__` option -With the `__temporality__` option, we could map monotonic deltas to the counter type, and non-monotonic counters to gauges. However, it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name), without adding [additional otel metric properties as labels](#add-otel-metric-properties-as-labels). +With the `__temporality__` option, we could map monotonic deltas to the counter type, and non-monotonic counters to gauges. However, it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Though this could be improved by adding [additional otel metric properties as labels](#add-otel-metric-properties-as-labels). ### Querying deltas alternatives From c1902f2d1b933bdde2eb4a33c3f38335d42a9e99 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 22 Jul 2025 17:42:14 +0100 Subject: [PATCH 26/34] Change to gauge + otel labels option Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 190 ++++++++---------- 1 file changed, 83 insertions(+), 107 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index a625644..6a1806b 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -56,6 +56,12 @@ This isn't true in all cases though, for example, the StatsD client libraries em [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) describes Chronosphere's experience of adding functionality to ingest OTEL delta metrics and query them back with PromQL, and there is additionally [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q). +### Monotonicity + +OTEL sums have a [monotonicity property](https://opentelemetry.io/docs/specs/otel/metrics/supplementary-guidelines/#monotonicity-property), which indicates if the sum can only increase (monotonic) or if it can increase and decrease (non-monotonic). Monotonic cumulative sums are mapped to Prometheus counters. Non-monotonic cumulative sums are mapped to Prometheus gauges, since Prometheus does not support counters that can decrease. This is because any drop in a Prometheus counter is assumed to be a counter reset. + +[StatsD counters are non-monotonic by definition](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), and [the StatsD receiver sets counters as non-monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). Since StatsD is so widely used, when considering delta support, it's important to make sure non-monotonic counters will work properly in Prometheus. + ### Pitfalls of the current solution #### Lack of out of order support @@ -110,93 +116,63 @@ This section outlines the proposed approach for delta temporality support. Addit ### Ingesting deltas +Add a feature flag for ingesting deltas: `--enable-feature=otlp-native-delta-ingestion`. + When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`. For the initial implementation, ignore `StartTimeUnixNano`. To ensure compatibility with the OTEL spec, this case should ideally be supported. A way to preserve `StartTimeUnixNano` is described in the potential future extension, [CT-per-sample](#ct-per-sample). -### Chunks - -For the initial implementation, reuse existing chunk encodings. - -Delta counters will use the standard XOR chunks for float samples. - -Delta histograms will use native histogram chunks with the `GaugeType` counter reset hint/header. The counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. A (bucket or total) count drop is detected as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Additionally, counter histogram chunks have the invariant that no count ever goes down baked into their implementation. `GaugeType` allows counts to go up and down, and does not cut new chunks on counter resets. - -### Delta metric type - -It is useful to be able to distinguish between delta and cumulative metrics. This would allow users to understand what the raw data represents and what functions are appropriate to use. Additionally, this could allow the query engine or UIs displaying Prometheus data apply different behaviour depending on the metric type to provide meaningful output. - -As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. +#### Set metric type as gauge -We propose to add two options as feature flags for ingesting deltas: +For this initial proposal, OTEL deltas will be treated as Prometheus gauges. When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. -1. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series. +A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. -2. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges. +Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. -We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but it brings Prometheus more in alignment with the OTEL model. The preferred approach from the Prometheus delta working group is the `__temporality__` option, but we need practical experience to validate this is actually the case. +This approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. -Below we explore the pros and cons of each option in more detail. +As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. If the `--enable-feature=type-and-unit-labels` (from PROM-39) is set, a `__type__="gauge"` will be added as a label. -#### Introduce `__temporality__` label +An alternative/potential future extension is to add a temporality property to the Prometheus counter type, discussed in [Introduce `__temporality__` label](#introduce-__temporality__-label). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. -This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. - -`--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on that feature. +#### Add otel metric properties as labels -When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`), and the `__temporality__="delta"` label will be added. As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. +The key problem of ingesting OTEL deltas as Prometheus gauges is the lack of distinction between OTEL gauges and OTEL deltas, even though their semantics differ. There are issues like: -Cumulative metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. - -**Pros** -* Clear distinction between OTEL delta metrics and OTEL gauge metrics when ingested into Prometheus, meaning users know the appriopiate functions to use for each type (e.g. `sum_over_time()` is unlikely to be useful for OTEL gauge metrics). -* Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having separate `__type__` and `__temporality__` labels in Prometheus. -* When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) +* Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. +* Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics. +* The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. -**Cons** -* Introduces additional complexity to the Prometheus data model. -* Confusing overlap between gauge and `__temporality__="delta"`. As mentioned in [Treat as gauge](#treat-as-gauge), essentially deltas already exist in Prometheus as gauges, and deltas can be viewed as a subset of gauges under the Prometheus definition. The same `sum_over_time()` would be used for aggregating these pre-existing deltas-as-gauges and OTEL deltas with counter type and `__temporality__="delta"`, creating confusion on why there are two different "types". - * Pre-existing deltas-as-gauges could be converted to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time. -* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. -* Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. +It is useful to be able to identify delta metrics and distinguish them from cumulative metrics and gauges. This would allow users to understand what the raw data represents and what functions are appropriate to use. Additionally, this could allow the query engine or UIs displaying Prometheus data apply different behaviour depending on the metric type to provide meaningful output. -#### Treat as gauge +Therefore it is important to maintain information about the OTEL metric properties. Alongside setting the type to `gauge` / `gaugehistogram`, the original OTEL metric properties will also be added as labels: -Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. +* `__otel_type__="sum"` +* `__otel_temporality__="delta"` +* `__otel_monotonicity__="true"/"false"` - as mentioned in [Monotonicity](#monotonicity), it is important to be able to ingest non-monotonic counters. Therefore this label is added to be able to distinguish between monotonic and non-monotonic cases. -Gauges ingested into Prometheus via scraping represent sampled values for the metric and using `sum_over_time()` for these types of gauges do not make sense. However, there are other sources that can ingest "delta" gauges already for which summing does make sense. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. +This is similar to the approach taken for type and unit labels in PROM-39. However, since these are new labels being added, there is not a strong dependency so does not require `--enable-feature=type-and-unit-labels` to be enabled. At query time, these should be dropped in the same way as the other metadata labels (as per PROM-39: "When a query drops the metric name in an effect of an operation or function, `__type__` and `__unit__` will also be dropped"), as these labels do provide type information about the metric and that changes when certain operations/functions are applied. -When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. +This approach makes delta support OTEL-specific, rather than a more generic Prometheus feature. However, Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may also be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. -**Pros** -* Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. -* Prometheus already uses gauges to represent deltas. For example, `increase()` outputs the delta of a counter series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. -* Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. +There is a risk that tooling and workflows could rely too much on the OTEL-specific labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also a separate type label to consider. -**Cons** -* Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge. -* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics. -* The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem. - -The cons are generally around it being difficult to tell apart OTEL gauges and counters when ingested into Prometheus. An extension could be to [Add otel metric properties as labels](#add-otel-metric-properties-as-labels), so there is extra information users can use to decide on how to query the metric, while the Prometheus type remains a gauge. - -### Metric names +#### Metric names OTEL metric names are normalised when translated to Prometheus by default ([code](https://github.com/prometheus/otlptranslator/blob/94f535e0c5880f8902ab8c7f13e572cfdcf2f18e/metric_namer.go#L157)). This includes adding suffixes in some cases. For example, OTEL metrics converted into Prometheus counters (i.e. monotonic cumulative sums in OTEL) have the `__total` suffix added to the metric name, while gauges do not. -The `_total` suffix will not be added to OTEL deltas, ingested as either counters with temporality label or gauges. The `_total` suffix is used to help users figure out whether a metric is a counter. As deltas depend on type and unit metadata labels being added, especially in the `--enable-feature=otlp-native-delta-ingestion` case, the `__type__` label will be able to provide the distinction and the suffix is unnecessary. +The `_total` suffix will not be added to OTEL deltas. The `_total` suffix is used to help users figure out whether a metric is a counter. The `__otel_type__` and `__otel_temporality__` labels will be able to provide the distinction and the suffix is unnecessary. This means switching between cumulative and delta temporality can result in metric names changing, affecting dashboards and alerts. However, the current proposal requires different functions for querying delta and cumulative counters anyway. -### Monotonicity - -OTEL sums have a [monotonicity property](https://opentelemetry.io/docs/specs/otel/metrics/supplementary-guidelines/#monotonicity-property), which indicates if the sum can only increase or if it can increase and decrease. Monotonic cumulative sums are mapped to Prometheus counters. Non-monotonic cumulative sums are mapped to Prometheus gauges, since Prometheus does not support counters that can decrease. This is because any drop in a Prometheus counter is assumed to be a counter reset. +#### Chunks -It is not necessary to detect counter resets for delta metrics - to get the increase over an interval, you can just sum the values over that interval. Therefore, for the `--enable-feature=otlp-native-delta-ingestion` option, where OTEL deltas are converted into Prometheus counters (with `__temporality__` label), non-monotonic delta sums will also be converted in the same way (with `__type__="counter"` and `__temporality__="delta"`). +For the initial implementation, reuse existing chunk encodings. -This ensures StatsD counters can be ingested as Prometheus counters. Since StatsD is so widely used, it's important to make sure StatsD counters (ingested via OTEL) will work properly in Prometheus. [StatsD counters are non-monotonic by definition](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), and [the StatsD receiver sets counters as non-monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). +Delta counters will use the standard XOR chunks for float samples. -Consequences include not being able to convert delta counters in Prometheus into their cumulative counterparts (e.g. for any possible future querying extensions for deltas). Also, as monoticity information is lost, if the metrics are later exported back into the OTEL format, all deltas will have to be assumed to be non-monotonic. +Delta histograms will use native histogram chunks with the `GaugeType` counter reset hint/header. The counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. A (bucket or total) count drop is detected as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Additionally, counter histogram chunks have the invariant that no count ever goes down baked into their implementation. `GaugeType` allows counts to go up and down, and does not cut new chunks on counter resets. ### Scraping @@ -206,17 +182,15 @@ No scraped metrics should have delta temporality as there is no additional benef Federating delta series directly could be usable if there is a constant and known collection interval for the delta series, and the metrics are scraped at least as often as the collection interval. However, this is not the case for all deltas and the scrape interval cannot be enforced. -Therefore we will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the `__temporality__="delta"` label is set. - -If deltas are converted to gauges, there would not be a way to distinguish deltas from regular gauges so we cannot provide a scrape config. +Therefore we will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the delta labels are set. ### Remote write ingestion -Remote write support is a non-goal for this initial delta proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because a label will be added to indicate the temporality of the metric and used during querying, and therefore can be added manually added to metrics before being sent by remote write. +Remote write support is a non-goal for this initial delta proposal to reduce its scope. However, the current design ends up supporting ingesting delta metrics via remote write. This is because OTEL labels are added to indicate the type and temporality of the metric, and therefore can be added manually added to metrics before being sent by remote write. ### Prometheus metric metadata -Prometheus has metric metadata as part of its metric model, which include the type of a metric. For this initial proposal, this will not be modified. Temporality will not be added as an additional metadata field, and will only be able to be set via the `__temporality__` label on a series. +Prometheus has metric metadata as part of its metric model, which include the type of a metric. This will not be modified for the initial proposal, since the OTEL type/temporality are just added as labels rather than extending the Prometheus metric model. ### Prometheus OTEL receivers @@ -224,11 +198,11 @@ Once deltas are ingested into Prometheus, they can be converted back into OTEL m The prometheusreceiver has the same issue described in [Scraping](#scraping) regarding possibly misaligned scrape vs delta collection intervals. -If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-native-delta-ingestion` will set the metric metadata type to counter. The receiver will currently assume it's a cumulative counter ([code](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7592debad2e93652412f2cd9eb299e9ac8d169f3/receiver/prometheusremotewritereceiver/receiver.go#L347-L351)), which is incorrect. If we gain more confidence that the `__temporality__` label is the correct approach, the receiver should be updated to translate counters with `__temporality__="delta"` to OTEL sums with delta temporality. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. +If we do not modify prometheusremotewritereceiver, then as the metric metadata type is set to gauge, the receiver will currently assume it's a OTEL gauge rather than an OTEL sum. If we gain more confidence that the OTEL labels is the correct approach, the receiver should be updated to translate Prometheus metrics based on the `__otel_...` labels. For now, we will recommend that delta metrics should be dropped before reaching the receiver, and provide a remote write relabel config for doing so. ### Querying deltas -For this initial proposal, existing functions will be used for querying deltas. This works for both the `__temporality__` and delta-as-gauge options. +For this initial proposal, existing functions will be used for querying deltas. `rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[]) / ` can be used for the rate. `metric / interval` can also be used to calculate a rate if the collection interval is known. @@ -268,10 +242,10 @@ Additionally, as mentioned before, it is common for deltas to have samples emitt #### Function warnings -To help users use the correct functions, warnings will be added if the metric type/temporality does not match the types that should be used with the function. +To help users use the correct functions, warnings will be added if the metric type/temporality does not match the types that should be used with the function. The warnings will be based on the `__type__` label, so will only work if `--enable-feature=type-and-unit-labels` is enabled. Because of this, while not necessary, it will be recommended to enabled the type and unit labels feature alongside the delta support feature. -* Cumulative counter-specific functions (`rate()`, `increase()`, `irate()` and `resets()`) will warn if `__type__="gauge"` or `__temporality__="delta"`. -* `sum_over_time()` will warn if `type="counter"` with no `__temporality__` label (implies cumulative counter), or if `__temporality__="cumulative"`. +* Cumulative counter-specific functions (`rate()`, `increase()`, `irate()` and `resets()`) will warn if `__type__="gauge"`. +* `sum_over_time()` will warn if `type="counter"`. ### Summary @@ -285,11 +259,9 @@ To work out the delta for all the cumulative native histograms in an range, the ## Implementation Plan -### 1. Experimental feature flags for OTLP delta ingestion - -Add the `--enable-feature=otlp-native-delta-ingestion` and `--enable-feature=otlp-delta-as-gauge-ingestion` feature flags as described in [Delta metric type](#delta-metric-type) to add appropriate types and flags to series on ingestion. +### 1. Experimental feature flag for OTLP delta ingestion -Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any type metadata to the ingested series. +A `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any type metadata to the ingested series. Update the feature to set the type to gauge and add the `__otel_...` labels. ### 2. Function warnings @@ -303,12 +275,40 @@ Update documentation explaining new experimental delta functionality, including ### 4. Review deltas in practice and experiment with possible future extensions -Review how deltas work in practice using the current approach, and use experience and feedback to decide whether any of the potential extensions are necessary, and whether to discontinue one of the two options for delta ingestion (gauges vs. temporality label). +Review how deltas work in practice using the current approach, and use experience and feedback to decide whether any of the potential extensions are necessary. ## Potential future extensions Potential extensions, some may require dedicated proposals. +### Introduce `__temporality__` label + +This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. + +`--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on that feature. + +When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`), and the `__temporality__="delta"` label will be added. As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. + +An additional `__monotonicity__` label will be added so both monotonic and non-monotonic deltas can be supported. If we just supported monotonic delta counters, StatsD counters would be rejected. If we wrote non-monotonic deltas as gauges (while keeping monotonic deltas as counters with `__temporality__`), the more advanced querying features we may implement in the future (e.g. allowing deltas to be queried with `rate()`) would not work with StatsD counters mapped to gauges. + +Cumulative metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. + +**Pros** +* Clear distinction between OTEL delta metrics and OTEL gauge metrics when ingested into Prometheus, meaning users know the appriopiate functions to use for each type (e.g. `sum_over_time()` is unlikely to be useful for OTEL gauge metrics). +* Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having separate `__type__` and `__temporality__` labels in Prometheus. +* When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) + +**Cons** +* Introduces additional complexity to the Prometheus data model. +* Confusing overlap between gauge and `__temporality__="delta"`. Essentially deltas already exist in Prometheus as gauges, and deltas can be viewed as a subset of gauges under the Prometheus definition. The same `sum_over_time()` would be used for aggregating these pre-existing deltas-as-gauges and OTEL deltas with counter type and `__temporality__="delta"`, creating confusion on why there are two different "types". + * Pre-existing deltas-as-gauges could be converted to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time. +* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. +* Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. + +We decided not to go for this approach for the initial version as it is more invasive to the Prometheus model - it changes the definition of a Prometheus counter, especially if we allow non-monotonic deltas to be ingested as counters. This would mean we won't always be able convert Prometheus delta counters to Prometheus cumulative counters, as Prometheus cumulative counters have to monotonic (as drops are detected as resets). There's no actual use case yet for being able to convert delta counters to cumulative ones but it makes the Prometheus data model more complex (counters sometimes need to be monotonic, but not always). + +With the `__otel_...` labels approach, the core Prometheus types do not change, just additional metadata that can help users decide how to write queries. However, if it seems like the `__otel_...` labels is too confusing, or there are use cases for delta temporality outside of OTEL, we may decide to change to this option. + ### CT-per-sample [CreatedTimestamp (PROM-29)](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md) records when a series was created or last reset, therefore allowing more accurate rate or increase calculations. This is similar to the `StartTimeUnixNano` field in OTEL metrics. @@ -321,6 +321,12 @@ Having CT-per-sample can improve the `rate()` calculation - the collection inter There are unknowns over the performance and storage of essentially doubling the number of samples with this approach. +An additional consideration would be any CreatedTimestamp features would need to work for both Prometheus counters and gauges, so that `StartTimeUnixNano` would be able to be preserved for deltas-as-gauges. + +### OTEL metric properties on all metrics + +For this initial proposal, the OTEL metric properties will only be added to delta metrics. If this works out, we should consider adding the properties to all metrics ingested via OTEL. This can be useful in other cases too, for example, distinguishing OTEL non-monotonic cumulative sums from OTEL gauges (which are currently both mapped to Prometheus gauges with no distinguishing labels). + ### Inject zeroes for StartTimeUnixNano [CreatedAt timestamps can be injected as 0-valued samples](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection). Similar could be done for StartTimeUnixNano. @@ -383,9 +389,9 @@ The numerous downsides likely outweigh the argument for consistency. Additionall ### Function overloading -`rate()` and `increase()` could be extended to work transparently with both cumulative and delta metrics. The PromQL engine could check the `__temporality__` label and execute the correct logic. +`rate()` and `increase()` could be extended to work transparently with both cumulative and delta metrics. The PromQL engine could check the `__otel_temporality__` label and execute the correct logic. -Function overloading could also work if OTEL deltas are ingested as Prometheus gauges and the `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on gauges that are unsuited to being summed over time, not add any warnings, and produce nonsensical results. +Function overloading could on all metrics where `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on gauges that are unsuited to being summed over time, not add any warnings, and produce nonsensical results. Pros: @@ -397,14 +403,14 @@ Cons: * The increased internal complexity could end up being more confusing. * Inconsistent functions depending on type and temporality. Both `increase()` and `sum_over_time()` could be used for aggregating deltas over time. However, `sum_over_time()` would not work for cumulative metrics, and `increase()` would not work for gauges. - * This gets further complicated when considering how functions could be combined. For example, a user may use recording rules to compute the sum of a delta metric over a shorter period of time, so they can later for the sum over a longer period of time faster by using the pre-computed results. `increase()` and `sum_over_time()` would both work for the recording rule. For the long query, `increase()` will not work - it would return a warning (as the pre-computed values would be gauges/untyped and should not be used with `increase()`) and a wrong value (if the cumulative counter logic is applied when there is no `__temporality__="delta"` label). `sum_over_time()` for the long query would work in the expected manner, however. + * This gets further complicated when considering how functions could be combined. For example, a user may use recording rules to compute the sum of a delta metric over a shorter period of time, so they can later for the sum over a longer period of time faster by using the pre-computed results. `increase()` and `sum_over_time()` would both work for the recording rule. For the longer query, `increase()` will not work - it would return a warning (as the pre-computed values would be gauges/untyped and should not be used with `increase()`) and a wrong value (if the cumulative counter logic is applied when there is no `__otel_temporality__="delta"` label). `sum_over_time()` for the longer query would work in the expected manner, however. * Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). * One possible extension could be to stitch the cumulative and delta series together and return a single result. * There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. Open questions and considerations: -* While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. +* While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__otel_temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. * There are open questions on how to best calculate the rate or increase of delta metrics (see [`rate()` behaviour for deltas](#rate-behaviour-for-deltas) below), and there is currently ongoing work with [extending range selectors for cumulative counters](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md), which should be taken into account for deltas too. * Once we start with overloading functions, users may ask for more of that e.g. should we change `sum_over_time()` to also allow calculating the increase of cumulative metrics rather than just summing samples together. Where would the line be in terms of which functions should be overloaded or not? One option would be to only allow `rate()` and `increase()` to be overloaded, as they are the most popular functions that would be used with counters. @@ -432,28 +438,6 @@ An alternative to function overloading, but allowing more choices on how rate ca This has the problem of having to use different functions for delta and cumulative metrics (switching cost, possibly poor user experience). -### `__monotonicity__` label - -A possible future enhancement is to add an `__monotonicity__` label along with `__temporality__` for counters. - -Additionally, if there were a reliable way to have [Created Timestamp](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md) for all cumulative counters, we could consider supporting non-monotonic cumulative counters as well, as at that point the created timestamp could be used for working out counter resets instead of decreases in counter value. This may not be feasible or wanted in all cases though. - -### Add otel metric properties as labels - -This would primarily be beneficial for the `--enable-feature=otlp-delta-as-gauge-ingestion` option. - -The key problem of ingesting OTEL deltas as Prometheus gauges is the lack of distinction between OTEL gauges and OTEL deltas, even though their semantics differ. Adding the original OTEL types as labels means that users would be able to get this information. For an OTEL delta sum, its labels would include `__type__="gauge"`, `__otel_type__="sum"` and `__otel_temporality__="delta"` (and `__otel_monotonicity__="true"/"false"`). This could also be useful for non-delta use cases, for example, being able to identify OTEL non-monotonic cumulative counters vs OTEL gauges. - -Function overloading could still be done - `rate()` and `increase()` could work based on the OTEL type label, rather than a general temporality label. - -This approach makes delta support OTEL-specific, rather than a more generic Prometheus feature. However, Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. - -There is a risk that tooling and workflows could rely too much on the OTEL-specific labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. - -This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also a separate type label to consider. - -An additional consideration would be any CreatedTimestamp features would need to work for both Prometheus counters and gauges, so that `StartTimeUnixNano` would be able to be preserved for deltas-as-gauges. - ## Discarded alternatives ### Ingesting deltas alternatives @@ -482,8 +466,6 @@ Users might want to convert back to original values (e.g. to sum the original va This also does not work for samples missing StartTimeUnixNano. -### Delta metric type alternatives - #### Add delta `__type__` label values Instead of a new `__temporality__` label, extend `__type__` from the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files) with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) @@ -496,12 +478,6 @@ Additionally, combining temporality and type means that every time a new type is Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. -### Monotonicity alternatives - -#### Map non-monotonic delta counters to gauges with `__temporality__` option - -With the `__temporality__` option, we could map monotonic deltas to the counter type, and non-monotonic counters to gauges. However, it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Though this could be improved by adding [additional otel metric properties as labels](#add-otel-metric-properties-as-labels). - ### Querying deltas alternatives #### Convert to cumulative on query From 9e0b8564ed1eac5057eb8a5450c09d6c1b318792 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Tue, 22 Jul 2025 20:23:58 +0100 Subject: [PATCH 27/34] Clarify otel_temporality vs temporality Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index 6a1806b..de173ec 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -134,7 +134,7 @@ This approach leverages an existing Prometheus metric type, reducing the changes As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. If the `--enable-feature=type-and-unit-labels` (from PROM-39) is set, a `__type__="gauge"` will be added as a label. -An alternative/potential future extension is to add a temporality property to the Prometheus counter type, discussed in [Introduce `__temporality__` label](#introduce-__temporality__-label). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. +An alternative/potential future extension is to add a temporality property to the Prometheus counter type, discussed in [Introduce `__temporality__` label](#ingest-as-counter-with-__temporality__-label). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. #### Add otel metric properties as labels @@ -154,7 +154,9 @@ Therefore it is important to maintain information about the OTEL metric properti This is similar to the approach taken for type and unit labels in PROM-39. However, since these are new labels being added, there is not a strong dependency so does not require `--enable-feature=type-and-unit-labels` to be enabled. At query time, these should be dropped in the same way as the other metadata labels (as per PROM-39: "When a query drops the metric name in an effect of an operation or function, `__type__` and `__unit__` will also be dropped"), as these labels do provide type information about the metric and that changes when certain operations/functions are applied. -This approach makes delta support OTEL-specific, rather than a more generic Prometheus feature. However, Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may also be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. +The temporality and monotonicity labels could added without the `otel_` prefix, since it doesn't clash with any reserved labels in Prometheus. However, this makes it clear that these concepts are OTEL-specific and are linked to the `__otel_type__`, rather than the `__type__` label for the Prometheus type. + +This approach makes delta support OTEL-specific, rather than a more generic Prometheus feature. Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may also be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. There is a risk that tooling and workflows could rely too much on the OTEL-specific labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also a separate type label to consider. @@ -281,20 +283,24 @@ Review how deltas work in practice using the current approach, and use experienc Potential extensions, some may require dedicated proposals. -### Introduce `__temporality__` label +### Ingest as counter with `__temporality__` label + +This option extends the metadata labels proposal (PROM-39). + +Ingest delta metrics as Prometheus counters. When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`). To be able to distinguish from cumulative counters, an additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. -This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. +The `__temporality__` label is more generic than the ingest-as-gauge option that uses a `__otel_temporality__` label, making temporality more of a "first-class citizen" within Prometheus, and enables deltas to be accepted from non-OTEL sources. -`--enable-feature=otlp-native-delta-ingestion` will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled, as it depends heavily on that feature. +This will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled. -When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`), and the `__temporality__="delta"` label will be added. As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. +As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. -An additional `__monotonicity__` label will be added so both monotonic and non-monotonic deltas can be supported. If we just supported monotonic delta counters, StatsD counters would be rejected. If we wrote non-monotonic deltas as gauges (while keeping monotonic deltas as counters with `__temporality__`), the more advanced querying features we may implement in the future (e.g. allowing deltas to be queried with `rate()`) would not work with StatsD counters mapped to gauges. +An additional `__monotonicity__` label will be added so both monotonic and non-monotonic deltas can be supported. If we just supported monotonic delta counters, StatsD counters would be rejected. If we wrote non-monotonic deltas as gauges (while keeping monotonic deltas as counters with `__temporality__`), the more advanced querying features we may implement in the future (e.g. allowing deltas to be queried with `rate()`) would not work with StatsD counters mapped to gauges. -Cumulative metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. +Cumulative counter-like metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. **Pros** -* Clear distinction between OTEL delta metrics and OTEL gauge metrics when ingested into Prometheus, meaning users know the appriopiate functions to use for each type (e.g. `sum_over_time()` is unlikely to be useful for OTEL gauge metrics). +* Clear distinction between OTEL delta metrics and OTEL gauge metrics when ingested into Prometheus, meaning users know the appropriate functions to use for each type (e.g. `sum_over_time()` is unlikely to be useful for OTEL gauge metrics). * Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having separate `__type__` and `__temporality__` labels in Prometheus. * When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) @@ -305,7 +311,7 @@ Cumulative metrics ingested via the OTLP endpoint will also have a `__temporalit * Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. -We decided not to go for this approach for the initial version as it is more invasive to the Prometheus model - it changes the definition of a Prometheus counter, especially if we allow non-monotonic deltas to be ingested as counters. This would mean we won't always be able convert Prometheus delta counters to Prometheus cumulative counters, as Prometheus cumulative counters have to monotonic (as drops are detected as resets). There's no actual use case yet for being able to convert delta counters to cumulative ones but it makes the Prometheus data model more complex (counters sometimes need to be monotonic, but not always). +We decided not to go for this approach for the initial version as it is more invasive to the Prometheus model - it changes the definition of a Prometheus counter, especially if we allow non-monotonic deltas to be ingested as counters. This would mean we won't always be able convert Prometheus delta counters to Prometheus cumulative counters, as Prometheus cumulative counters have to monotonic (as drops are detected as resets). There's no actual use case yet for being able to convert delta counters to cumulative ones but it makes the Prometheus data model more complex (counters sometimes need to be monotonic, but not always). With the `__otel_...` labels approach, the core Prometheus types do not change, just additional metadata that can help users decide how to write queries. However, if it seems like the `__otel_...` labels is too confusing, or there are use cases for delta temporality outside of OTEL, we may decide to change to this option. From 79224c817b6f3ada4fbcd3d15dc9cf32b623952b Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Wed, 23 Jul 2025 10:34:35 +0100 Subject: [PATCH 28/34] Justify not having delta_counter type better Signed-off-by: Fiona Liao --- proposals/0048-otel_delta_temporality_support.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index de173ec..7f337d3 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -130,11 +130,11 @@ A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.i Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. -This approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model. +This approach leverages an existing Prometheus metric type, avoiding the changes to the core Prometheus data model. As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. If the `--enable-feature=type-and-unit-labels` (from PROM-39) is set, a `__type__="gauge"` will be added as a label. -An alternative/potential future extension is to add a temporality property to the Prometheus counter type, discussed in [Introduce `__temporality__` label](#ingest-as-counter-with-__temporality__-label). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. +An alternative/potential future extension is to add a temporality property to the Prometheus counter type and allow OTEL deltas to be ingested as Prometheus counters, discussed in [Ingest as counter with `__temporality__` label](#ingest-as-counter-with-__temporality__-label). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. #### Add otel metric properties as labels @@ -410,7 +410,7 @@ Cons: * The increased internal complexity could end up being more confusing. * Inconsistent functions depending on type and temporality. Both `increase()` and `sum_over_time()` could be used for aggregating deltas over time. However, `sum_over_time()` would not work for cumulative metrics, and `increase()` would not work for gauges. * This gets further complicated when considering how functions could be combined. For example, a user may use recording rules to compute the sum of a delta metric over a shorter period of time, so they can later for the sum over a longer period of time faster by using the pre-computed results. `increase()` and `sum_over_time()` would both work for the recording rule. For the longer query, `increase()` will not work - it would return a warning (as the pre-computed values would be gauges/untyped and should not be used with `increase()`) and a wrong value (if the cumulative counter logic is applied when there is no `__otel_temporality__="delta"` label). `sum_over_time()` for the longer query would work in the expected manner, however. -* Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). +* Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__otel_temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). * One possible extension could be to stitch the cumulative and delta series together and return a single result. * There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. @@ -474,12 +474,18 @@ This also does not work for samples missing StartTimeUnixNano. #### Add delta `__type__` label values -Instead of a new `__temporality__` label, extend `__type__` from the [proposal to add type and unit metadata labels to metrics](https://github.com/prometheus/proposals/pull/39/files) with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. (Note: type metadata might become native series information rather than labels; if that happens, we'd use that for indicating the delta types instead of labels.) +Extend `__type__` from PROM-39 with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. -A downside is that querying for all counter types or all delta series is less efficient - regex matchers like `__type__=~”(delta_counter|counter)”` or `__type__=~”delta_.*”` would have to be used. (However, this does not seem like a particularly necessary use case to optimise for.) +This would prevent the problems of mixups with the `counter` type that [Ingest as counter with `__temporality__` label](#ingest-as-counter-with-__temporality__-label) has as it's a completely new type. + +An additional `__monotonicity__` property would need to be added as well. + +A downside is that querying for all counter types or all delta series is less efficient compared to the `__temporality__` case - regex matchers like `__type__=~”(delta_counter|counter)”` or `__type__=~”delta_.*”` would have to be used. (However, this does not seem like a particularly necessary use case to optimise for.) Additionally, combining temporality and type means that every time a new type is added to Prometheus/OTEL, two `__type__` values would have to be added. This is unlikely to happen very often, so only a minor con. +This option was discarded because, for the initial proposal, we want to reuse the existing gauge type in Prometheus (since it is already used for deltas) instead of introducing changes to the Prometheus data model. If we later decide to extend the Prometheus types, it is better to treat temporality as a property of a counter rather than as part of the type itself, since temporality is an property that can apply to multiple metric types (for both counter and histograms). + #### Metric naming convention Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name. From 563104bb5cbf680e29f6c48118b467f5829bef17 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Wed, 23 Jul 2025 13:13:46 +0100 Subject: [PATCH 29/34] Clarifications Signed-off-by: Fiona Liao --- proposals/0048-otel_delta_temporality_support.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index 7f337d3..de77809 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -124,11 +124,11 @@ For the initial implementation, ignore `StartTimeUnixNano`. To ensure compatibil #### Set metric type as gauge -For this initial proposal, OTEL deltas will be treated as Prometheus gauges. When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. +For this initial proposal, OTEL deltas will be treated as a special case of Prometheus gauges. When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. -Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. +Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. Another way of putting it is that deltas are already a subset of the Prometheus gauge type. This approach leverages an existing Prometheus metric type, avoiding the changes to the core Prometheus data model. @@ -136,7 +136,7 @@ As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_typ An alternative/potential future extension is to add a temporality property to the Prometheus counter type and allow OTEL deltas to be ingested as Prometheus counters, discussed in [Ingest as counter with `__temporality__` label](#ingest-as-counter-with-__temporality__-label). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. -#### Add otel metric properties as labels +#### Add otel metric properties as informational labels The key problem of ingesting OTEL deltas as Prometheus gauges is the lack of distinction between OTEL gauges and OTEL deltas, even though their semantics differ. There are issues like: @@ -206,7 +206,7 @@ If we do not modify prometheusremotewritereceiver, then as the metric metadata t For this initial proposal, existing functions will be used for querying deltas. -`rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[]) / ` can be used for the rate. `metric / interval` can also be used to calculate a rate if the collection interval is known. +`rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[]) / ` can be used for the rate. A single delta sample can also be meaningful without aggregating over time as unlike cumulative metrics, each delta sample is independent. `metric / interval` can also be used to calculate a rate if the collection interval is known. Having different functions for delta and cumulative counters mean that if the temporality of a metric changes, queries will have to be updated. @@ -285,7 +285,7 @@ Potential extensions, some may require dedicated proposals. ### Ingest as counter with `__temporality__` label -This option extends the metadata labels proposal (PROM-39). +This option is a more generic solution to adding deltas to Prometheus, modifying the Prometheus model to allow counters to have either cumulative or delta temporality. It extends the metadata labels proposal (PROM-39). Ingest delta metrics as Prometheus counters. When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`). To be able to distinguish from cumulative counters, an additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. @@ -311,7 +311,7 @@ Cumulative counter-like metrics ingested via the OTLP endpoint will also have a * Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. -We decided not to go for this approach for the initial version as it is more invasive to the Prometheus model - it changes the definition of a Prometheus counter, especially if we allow non-monotonic deltas to be ingested as counters. This would mean we won't always be able convert Prometheus delta counters to Prometheus cumulative counters, as Prometheus cumulative counters have to monotonic (as drops are detected as resets). There's no actual use case yet for being able to convert delta counters to cumulative ones but it makes the Prometheus data model more complex (counters sometimes need to be monotonic, but not always). +We decided not to go for this approach for the initial version as it is more invasive to the Prometheus model - it changes the definition of a Prometheus counter, especially if we allow non-monotonic deltas to be ingested as counters. This would mean we won't always be able convert Prometheus delta counters to Prometheus cumulative counters, as Prometheus cumulative counters have to monotonic (as drops are detected as resets). There's no actual use case yet for being able to convert delta counters to cumulative ones but it makes the Prometheus data model more complex (counters sometimes need to be monotonic, but not always). An alternative is allow cumulative counters to be non-monotonic (by setting a `__monontonicity__="false"` label) and adding warnings if a non-monotonic cumulative counter is used with `rate()`, but that makes `rate()` usages more confusing. With the `__otel_...` labels approach, the core Prometheus types do not change, just additional metadata that can help users decide how to write queries. However, if it seems like the `__otel_...` labels is too confusing, or there are use cases for delta temporality outside of OTEL, we may decide to change to this option. From 0deadb86224d1598ca0c817f5f4eefbe44aec4be Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Thu, 24 Jul 2025 18:34:57 +0100 Subject: [PATCH 30/34] Amend post dev summit (drop otel prefix from label names) Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 52 ++++++++----------- 1 file changed, 22 insertions(+), 30 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index de77809..cd5625d 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -128,13 +128,13 @@ For this initial proposal, OTEL deltas will be treated as a special case of Prom A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. -Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. Another way of putting it is that deltas are already a subset of the Prometheus gauge type. +Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over a specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. Another way of putting it is that deltas are already a subset of the Prometheus gauge type. This approach leverages an existing Prometheus metric type, avoiding the changes to the core Prometheus data model. As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. If the `--enable-feature=type-and-unit-labels` (from PROM-39) is set, a `__type__="gauge"` will be added as a label. -An alternative/potential future extension is to add a temporality property to the Prometheus counter type and allow OTEL deltas to be ingested as Prometheus counters, discussed in [Ingest as counter with `__temporality__` label](#ingest-as-counter-with-__temporality__-label). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. +An alternative/potential future extension is to [ingest as Prometheus counters instead](#ingest-as-counter). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. #### Add otel metric properties as informational labels @@ -148,23 +148,23 @@ It is useful to be able to identify delta metrics and distinguish them from cumu Therefore it is important to maintain information about the OTEL metric properties. Alongside setting the type to `gauge` / `gaugehistogram`, the original OTEL metric properties will also be added as labels: -* `__otel_type__="sum"` -* `__otel_temporality__="delta"` -* `__otel_monotonicity__="true"/"false"` - as mentioned in [Monotonicity](#monotonicity), it is important to be able to ingest non-monotonic counters. Therefore this label is added to be able to distinguish between monotonic and non-monotonic cases. +* `__otel_type__="sum"` - this will allow the metric to be converted back to an OTEL sum (with delta temporality) rather than a gauge if the metrics as exported back to OTEL. +* `__temporality__="delta"` +* `__monotonicity__="true"/"false"` - as mentioned in [Monotonicity](#monotonicity), it is important to be able to ingest non-monotonic counters. Therefore this label is added to be able to distinguish between monotonic and non-monotonic cases. -This is similar to the approach taken for type and unit labels in PROM-39. However, since these are new labels being added, there is not a strong dependency so does not require `--enable-feature=type-and-unit-labels` to be enabled. At query time, these should be dropped in the same way as the other metadata labels (as per PROM-39: "When a query drops the metric name in an effect of an operation or function, `__type__` and `__unit__` will also be dropped"), as these labels do provide type information about the metric and that changes when certain operations/functions are applied. +The temporality and monotonicity labels do not have the `otel_` prefix, since they don't clash with any reserved labels in Prometheus. -The temporality and monotonicity labels could added without the `otel_` prefix, since it doesn't clash with any reserved labels in Prometheus. However, this makes it clear that these concepts are OTEL-specific and are linked to the `__otel_type__`, rather than the `__type__` label for the Prometheus type. +This is similar to the approach taken for type and unit labels in PROM-39. However, since these are new labels being added, there is not a strong dependency so does not require `--enable-feature=type-and-unit-labels` to be enabled. At query time, these should be dropped in the same way as the other metadata labels (as per PROM-39: "When a query drops the metric name in an effect of an operation or function, `__type__` and `__unit__` will also be dropped"), as these labels do provide type information about the metric and that changes when certain operations/functions are applied. -This approach makes delta support OTEL-specific, rather than a more generic Prometheus feature. Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may also be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. +This approach makes delta support more OTEL-specific, rather than a more generic Prometheus feature. Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may also be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. -There is a risk that tooling and workflows could rely too much on the OTEL-specific labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also a separate type label to consider. +There is a risk that tooling and workflows could rely too much on the OTEL-added labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also a separate type label to consider. #### Metric names OTEL metric names are normalised when translated to Prometheus by default ([code](https://github.com/prometheus/otlptranslator/blob/94f535e0c5880f8902ab8c7f13e572cfdcf2f18e/metric_namer.go#L157)). This includes adding suffixes in some cases. For example, OTEL metrics converted into Prometheus counters (i.e. monotonic cumulative sums in OTEL) have the `__total` suffix added to the metric name, while gauges do not. -The `_total` suffix will not be added to OTEL deltas. The `_total` suffix is used to help users figure out whether a metric is a counter. The `__otel_type__` and `__otel_temporality__` labels will be able to provide the distinction and the suffix is unnecessary. +The `_total` suffix will not be added to OTEL deltas. The `_total` suffix is used to help users figure out whether a metric is a counter. The `__otel_type__` and `__temporality__` labels will be able to provide the distinction and the suffix is unnecessary. This means switching between cumulative and delta temporality can result in metric names changing, affecting dashboards and alerts. However, the current proposal requires different functions for querying delta and cumulative counters anyway. @@ -283,37 +283,29 @@ Review how deltas work in practice using the current approach, and use experienc Potential extensions, some may require dedicated proposals. -### Ingest as counter with `__temporality__` label - -This option is a more generic solution to adding deltas to Prometheus, modifying the Prometheus model to allow counters to have either cumulative or delta temporality. It extends the metadata labels proposal (PROM-39). - -Ingest delta metrics as Prometheus counters. When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and thus the `__type__` label will be `counter` / `histogram`). To be able to distinguish from cumulative counters, an additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative. - -The `__temporality__` label is more generic than the ingest-as-gauge option that uses a `__otel_temporality__` label, making temporality more of a "first-class citizen" within Prometheus, and enables deltas to be accepted from non-OTEL sources. +### Ingest as counter -This will only be allowed to be enabled if `--enable-feature=type-and-unit-labels` is also enabled. +Instead of ingesting as a Prometheus gauge, ingest as a Prometheus counter. When ingesting a delta metric via the OTLP endpoint, the metric type is set to `counter` / `histogram` (and with PROM-39, the `__type__` label will be `counter` / `histogram`). -As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. +This will change the Prometheus definition of a counter (currently: "A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart"). -An additional `__monotonicity__` label will be added so both monotonic and non-monotonic deltas can be supported. If we just supported monotonic delta counters, StatsD counters would be rejected. If we wrote non-monotonic deltas as gauges (while keeping monotonic deltas as counters with `__temporality__`), the more advanced querying features we may implement in the future (e.g. allowing deltas to be queried with `rate()`) would not work with StatsD counters mapped to gauges. +As mentioned in the [Chunks](#chunks) section, `GaugeType` should still be the counter reset hint/header. Cumulative counter-like metrics ingested via the OTLP endpoint will also have a `__temporality__="cumulative"` label added. **Pros** * Clear distinction between OTEL delta metrics and OTEL gauge metrics when ingested into Prometheus, meaning users know the appropriate functions to use for each type (e.g. `sum_over_time()` is unlikely to be useful for OTEL gauge metrics). -* Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having separate `__type__` and `__temporality__` labels in Prometheus. -* When instrumenting with the OTEL SDK, the type needs to be explicitly defined for a metric but not its temporality. Additionally, the temporality of metrics could change in the metric processing pipeline (for example, using the deltatocumulative or cumulativetodelta processors). As a result, users may know the type of a metric but be unaware of its temporality at query time. If different query functions are required for delta versus cumulative metrics, it is difficult to know which one to use. By representing both type and temporality as metadata, there is the potential for functions like `rate()` to be overloaded or adapted to handle any counter-like metric correctly, regardless of its temporality. (See [Function overloading](#function-overloading) for more discussion.) +* Closer match with the OTEL model - in OTEL, counter-like types sum over events over time, with temporality being an property of the type. This is mirrored by having `__type__="counter"` and `__temporality__="delta"/"cumulative"` labels in Prometheus. **Cons** * Introduces additional complexity to the Prometheus data model. -* Confusing overlap between gauge and `__temporality__="delta"`. Essentially deltas already exist in Prometheus as gauges, and deltas can be viewed as a subset of gauges under the Prometheus definition. The same `sum_over_time()` would be used for aggregating these pre-existing deltas-as-gauges and OTEL deltas with counter type and `__temporality__="delta"`, creating confusion on why there are two different "types". +* Confusing overlap between gauge and delta counters. Essentially deltas already exist in Prometheus as gauges, and deltas can be viewed as a subset of gauges under the Prometheus definition. The same `sum_over_time()` would be used for aggregating these pre-existing deltas-as-gauges and delta counters, creating confusion on why there are two different "types". * Pre-existing deltas-as-gauges could be converted to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time. -* Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement. * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors. We decided not to go for this approach for the initial version as it is more invasive to the Prometheus model - it changes the definition of a Prometheus counter, especially if we allow non-monotonic deltas to be ingested as counters. This would mean we won't always be able convert Prometheus delta counters to Prometheus cumulative counters, as Prometheus cumulative counters have to monotonic (as drops are detected as resets). There's no actual use case yet for being able to convert delta counters to cumulative ones but it makes the Prometheus data model more complex (counters sometimes need to be monotonic, but not always). An alternative is allow cumulative counters to be non-monotonic (by setting a `__monontonicity__="false"` label) and adding warnings if a non-monotonic cumulative counter is used with `rate()`, but that makes `rate()` usages more confusing. -With the `__otel_...` labels approach, the core Prometheus types do not change, just additional metadata that can help users decide how to write queries. However, if it seems like the `__otel_...` labels is too confusing, or there are use cases for delta temporality outside of OTEL, we may decide to change to this option. +We may switch to this option if marking as the type as gauge appears too confusing to users. ### CT-per-sample @@ -395,9 +387,9 @@ The numerous downsides likely outweigh the argument for consistency. Additionall ### Function overloading -`rate()` and `increase()` could be extended to work transparently with both cumulative and delta metrics. The PromQL engine could check the `__otel_temporality__` label and execute the correct logic. +`rate()` and `increase()` could be extended to work transparently with both cumulative and delta metrics. The PromQL engine could check the `__temporality__="delta"` label and specific delta logic, otherwise treat as a cumulative counter. -Function overloading could on all metrics where `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on gauges that are unsuited to being summed over time, not add any warnings, and produce nonsensical results. +Function overloading could work on all metrics where `__type__="gauge"` label is added, but then `rate()` and `increase()` could run on gauges that are unsuited to being summed over time, not add any warnings, and produce nonsensical results. Pros: @@ -409,14 +401,14 @@ Cons: * The increased internal complexity could end up being more confusing. * Inconsistent functions depending on type and temporality. Both `increase()` and `sum_over_time()` could be used for aggregating deltas over time. However, `sum_over_time()` would not work for cumulative metrics, and `increase()` would not work for gauges. - * This gets further complicated when considering how functions could be combined. For example, a user may use recording rules to compute the sum of a delta metric over a shorter period of time, so they can later for the sum over a longer period of time faster by using the pre-computed results. `increase()` and `sum_over_time()` would both work for the recording rule. For the longer query, `increase()` will not work - it would return a warning (as the pre-computed values would be gauges/untyped and should not be used with `increase()`) and a wrong value (if the cumulative counter logic is applied when there is no `__otel_temporality__="delta"` label). `sum_over_time()` for the longer query would work in the expected manner, however. -* Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__otel_temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). + * This gets further complicated when considering how functions could be combined. For example, a user may use recording rules to compute the sum of a delta metric over a shorter period of time, so they can later for the sum over a longer period of time faster by using the pre-computed results. `increase()` and `sum_over_time()` would both work for the recording rule. For the longer query, `increase()` will not work - it would return a warning (as the pre-computed values would be gauges/untyped and should not be used with `increase()`) and a wrong value (if the cumulative counter logic is applied when there is no `__temporality__="delta"` label). `sum_over_time()` for the longer query would work in the expected manner, however. +* Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). * One possible extension could be to stitch the cumulative and delta series together and return a single result. * There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. Open questions and considerations: -* While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__otel_temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. +* While there is some precedent for function overloading with both counters and native histograms being processed in different ways by `rate()`, those are established types with obvious structual differences that are difficult to mix up. The metadata labels (including the proposed `__temporality__` label) are themselves experimental and require more adoption and validation before we start building too much on top of them. * There are open questions on how to best calculate the rate or increase of delta metrics (see [`rate()` behaviour for deltas](#rate-behaviour-for-deltas) below), and there is currently ongoing work with [extending range selectors for cumulative counters](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md), which should be taken into account for deltas too. * Once we start with overloading functions, users may ask for more of that e.g. should we change `sum_over_time()` to also allow calculating the increase of cumulative metrics rather than just summing samples together. Where would the line be in terms of which functions should be overloaded or not? One option would be to only allow `rate()` and `increase()` to be overloaded, as they are the most popular functions that would be used with counters. From 4a73811c5830cae394451841e3b9f386dd4a52ab Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Thu, 24 Jul 2025 19:04:52 +0100 Subject: [PATCH 31/34] Point out that no otel prefix means it starts becoming a generic label Signed-off-by: Fiona Liao --- proposals/0048-otel_delta_temporality_support.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index cd5625d..9b9064a 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -152,13 +152,11 @@ Therefore it is important to maintain information about the OTEL metric properti * `__temporality__="delta"` * `__monotonicity__="true"/"false"` - as mentioned in [Monotonicity](#monotonicity), it is important to be able to ingest non-monotonic counters. Therefore this label is added to be able to distinguish between monotonic and non-monotonic cases. -The temporality and monotonicity labels do not have the `otel_` prefix, since they don't clash with any reserved labels in Prometheus. - This is similar to the approach taken for type and unit labels in PROM-39. However, since these are new labels being added, there is not a strong dependency so does not require `--enable-feature=type-and-unit-labels` to be enabled. At query time, these should be dropped in the same way as the other metadata labels (as per PROM-39: "When a query drops the metric name in an effect of an operation or function, `__type__` and `__unit__` will also be dropped"), as these labels do provide type information about the metric and that changes when certain operations/functions are applied. -This approach makes delta support more OTEL-specific, rather than a more generic Prometheus feature. Prometheus does not natively produce deltas anyway as discussed in [Scraping](#scraping) (though users could push deltas to Prometheus via remote write). It may also be preferable to acknowledge the fundamental differences between the OTEL and Prometheus metric models, and treat OTEL deltas as a special case for compatibility, instead of integrating them as a core Prometheus feature. +The temporality and monotonicity labels do not have the `otel_` prefix, since they don't clash with any reserved labels in Prometheus. This means that metrics ingested via other sources (e.g. remote write) could add `__temporality__="delta"` and `__monotonicity__="true"/"false"` labels too as a way of ingesting delta metrics. -There is a risk that tooling and workflows could rely too much on the OTEL-added labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also a separate type label to consider. +There is a risk that tooling and workflows could rely too much on the OTEL-added labels rather than Prometheus native types where possible, leading to inconsistent user experience depending on whether OTEL or Prometheus is used for ingestion. This can also be confusing, as OTEL users would need to understand there are two different types - the OTEL types and the Prometheus types. For example, knowing that an OTEL gauge is not the same as a Prometheus gauge and there's also additional labels to consider. #### Metric names From fa69351679ee084a9160ecd4b093b1105aeda926 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 8 Aug 2025 15:14:57 +0100 Subject: [PATCH 32/34] Elaborate on usefulness of CT-per-sample Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 44 ++++++++++++------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index 9b9064a..324a164 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -208,7 +208,7 @@ For this initial proposal, existing functions will be used for querying deltas. Having different functions for delta and cumulative counters mean that if the temporality of a metric changes, queries will have to be updated. -Possible improvements to rate/increase calculations and user experience can be found in [Rate calculation extensions](#rate-calculation-extensions) and [Function overloading](#function-overloading). +Possible improvements to rate/increase calculations and user experience can be found in [Rate calculation extensions](#rate-calculation-extensions-without-ct-per-sample) and [Function overloading](#function-overloading). Note: With [left-open range selectors](https://prometheus.io/docs/prometheus/3.5/migration/#range-selectors-and-lookback-exclude-samples-coinciding-with-the-left-boundary) introduced in Prometheus 3.0, queries such as `sum_over_time(metric[])` will exclude the sample at the left boundary. This is a fortunate usability improvement for querying deltas - with Prometheus 2, a `1m` interval actually covered `1m1s`, which could lead to double counting samples in consecutive steps and inflated sums; to get the actual value within `1m`, the awkward `59s999ms` had to be used instead. @@ -315,9 +315,7 @@ CT-per-sample is not a blocker for deltas - before this is ready, `StartTimeUnix Having CT-per-sample can improve the `rate()` calculation - the collection interval for each sample will be directly available, rather than having to guess the interval based on gaps. It also means a single sample in the range can result in a result from `rate()` as the range will effectively have an additional point at `StartTimeUnixNano`. -There are unknowns over the performance and storage of essentially doubling the number of samples with this approach. - -An additional consideration would be any CreatedTimestamp features would need to work for both Prometheus counters and gauges, so that `StartTimeUnixNano` would be able to be preserved for deltas-as-gauges. +There are unknowns over the performance and storage with this approach. ### OTEL metric properties on all metrics @@ -333,12 +331,14 @@ If CT-per-sample takes too long, this could be a temporary solution. It's possible for the StartTimeUnixNano of a sample to be the same as the TimeUnixNano of the preceding sample; care would need to be taken to not overwrite the non-zero sample value. -### Rate calculation extensions +### Rate calculation extensions (without CT-per-sample) [Querying deltas](#querying-deltas) outlined the caveats of using `sum_over_time(...[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. This section assumes knowledge of [Extended range selectors semantics proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md) which introduces the `smoothed` and `anchored` modifers to range selectors, in particular for `rate()` and `increase()` for cumulative counters. +This also assumes CT-per-sample is not available. A possibly simpler solution if CT-per-sample is available is discussed in [Treat as mini-cumulative](#treat-as-mini-cumulative). + ##### Lookahead and lookbehind of range The reason why `increase()`/`rate()` need extrapolation for cumulative counters is to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. @@ -419,34 +419,44 @@ Also to take into account are the new `smoothed` and `anchored` modifiers in the A possible proposal would be: * no modifier - just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). -* `anchored` - same as no modifer. In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas. To get the same output in the cumulative and delta cases given the same input to the initial instrumented counter, the delta case should use `sum_over_time()`. +* `anchored` - same as no modifer. In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas (if CT-per-sample is not implemented). To get the same output in the cumulative and delta cases given the same input to the initial instrumented counter, the delta case should use `sum_over_time()`. * `smoothed` - Logic as described in [Lookahead and lookbehind](#lookahead-and-lookbehind-of-range). One problem with reusing the range selector modifiers is that they are more generic than just modifiers for `rate()` and `increase()`, so adding delta-specific logic for these modifiers for `rate()` and `increase()` may be confusing. -#### How to proceed - -Before committing to moving forward with function overloading, we should first gain practical experience with the use of `sum_over_time()` for delta metrics and see if there's a real need for overloading, and observe how the `smoothed` and `anchored` modifiers work in practice for cumulative metrics. - ### `delta_*` functions An alternative to function overloading, but allowing more choices on how rate calculation can be done would be to introduce `delta_*` functions like `delta_rate()` and having range selector modifiers. This has the problem of having to use different functions for delta and cumulative metrics (switching cost, possibly poor user experience). -## Discarded alternatives - -### Ingesting deltas alternatives - -#### Treat as “mini-cumulative” +### Treat as “mini-cumulative” Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. This requires CT-per-sample (or some kind of precise CT tracking) to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. -Functions will not take into account delta-specific characteristics. The OTEL SDKs only emit datapoints when there is a change in the interval. `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate, which is not always true for delta samples. +The standard `rate()`/`increase()` functions do not work well with delta-specific characteristics, especially without CT-per-sample. The OTEL SDKs only emit datapoints when there is a change in the interval so samples might not come at consistent intervals. But `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate. -This also does not work for samples missing StartTimeUnixNano. +With CT-per-sample, `rate()` can make this information to more accurately calculate the intervals between samples. Assuming `rate()` continues to be constrained to only looking at the samples with timestamps within its bounds though, it will still not be able to accurately predict whether to extrapolate at the end of the time range is. The reason for extrapolating is that there might be a sample with timestamp outside of the range, but its CT is within the range of the query. With sparse delta series which only push updates when there is a change, a long gap between samples does not mean the series is definitively ending. + +However, the experimental `anchored` and `smoothed` modifiers could be used in conjunction with CT-per-sample to be able to calculate `rate()` without needing to extrapolate (and so does not encounter the issue not being able to accurately guess when a series ends). The improvement of having CT-per-sample over [extending these modifiers but without CT-per sample](#rate-calculation-extensions-without-ct-per-sample) is that, aside from not needing delta special-casing, is that `smoothed` has better information about the actual interval for samples rather than having to assume samples are evenly spaced and continuous. + +For samples missing missing StartTimeUnixNano, the CT can be set equal to the sample timestamp, indicating the value was recorded for a single instant in time. `rate()` might be able to work with this without any special adjustments (it would see sample interval is equal to 0 and this would stop any extrapolation). + +The benefit of this "mini-cumulative" approach is that we would not need to introduce a new type (or subtype) for deltas, the current counter type definition would be sufficient for monotonic deltas. Additionally, `rate()` could be used without needing function overloading - deltas will be operated on as regular cumulative counters without special casing. + +Additionally, having CT-per-sample could allow for non-monotonic OTELs sums for both cumulative and delta temporality to be supported as counters (as we will not need to use drops in the metric value to detect counter resets). + +In comparison to the initial proposal: The distinguishing labels suggested in the initial proposal (like `__temporality__`) would not be necessary in terms of querying, however, they could be kept for enabling translation back into OTEL. Knowing the temporality could also be useful for understanding the raw data. `sum_over_time()` would still be able to be used on the ingested delta metrics. `__type__` would change from `gauge` to `counter`. + +### How to proceed + +Before committing to moving forward with any of the extensions , we should first gain practical experience with treating delta metrics as gauges and using `sum_over_time()` to query, and observe how the `smoothed` and `anchored` modifiers work in practice for cumulative metrics. + +## Discarded alternatives + +### Ingesting deltas alternatives #### Convert to rate on ingest From 986a3121c8fb935f8546c0b4bb46dd7c21c66436 Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 12 Sep 2025 18:31:47 +0100 Subject: [PATCH 33/34] Fix links and format Signed-off-by: Fiona Liao --- .../0048-otel_delta_temporality_support.md | 75 +++++++++---------- 1 file changed, 37 insertions(+), 38 deletions(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index 324a164..76a6c0a 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -1,4 +1,3 @@ - # OTEL delta temporality support * **Owners:** @@ -26,9 +25,9 @@ A proposal for adding experimental support for OTEL delta temporality metrics in Prometheus supports the ingestion of OTEL metrics via its OTLP endpoint. Counter-like OTEL metrics (e.g. histograms, sum) can have either [cumulative or delta temporality](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality). However, Prometheus only supports cumulative metrics, due to its pull-based approach to collecting metrics. -Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. +Therefore, delta metrics need to be converted to cumulative ones during ingestion. The OTLP endpoint in Prometheus has an [experimental feature to convert delta to cumulative](https://github.com/prometheus/prometheus/blob/9b4c8f6be28823c604aab50febcd32013aa4212c/docs/feature_flags.md?plain=1#L167). Alternatively, users can run the [deltatocumulative processor](https://github.com/sh0rez/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in their OTEL pipeline before writing the metrics to Prometheus. -The cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the [Pitfalls section](#pitfalls-of-the-current-solution) below. +The cumulative code for storage and querying can be reused, and when querying, users don’t need to think about the temporality of the metrics - everything just works. However, there are downsides elaborated in the [Pitfalls section](#pitfalls-of-the-current-solution) below. Prometheus' goal of becoming the best OTEL metrics backend means it should improve its support for delta metrics, allowing them to be ingested and stored without being transformed into cumulative. @@ -56,11 +55,11 @@ This isn't true in all cases though, for example, the StatsD client libraries em [Chronosphere Delta Experience Report](https://docs.google.com/document/d/1L8jY5dK8-X3iEoljz2E2FZ9kV2AbCa77un3oHhariBc/edit?tab=t.0#heading=h.3gflt74cpc0y) describes Chronosphere's experience of adding functionality to ingest OTEL delta metrics and query them back with PromQL, and there is additionally [Musings on delta temporality in Prometheus](https://docs.google.com/document/d/1vMtFKEnkxRiwkr0JvVOrUrNTogVvHlcEWaWgZIqsY7Q/edit?tab=t.0#heading=h.5sybau7waq2q). -### Monotonicity +### Monotonicity OTEL sums have a [monotonicity property](https://opentelemetry.io/docs/specs/otel/metrics/supplementary-guidelines/#monotonicity-property), which indicates if the sum can only increase (monotonic) or if it can increase and decrease (non-monotonic). Monotonic cumulative sums are mapped to Prometheus counters. Non-monotonic cumulative sums are mapped to Prometheus gauges, since Prometheus does not support counters that can decrease. This is because any drop in a Prometheus counter is assumed to be a counter reset. -[StatsD counters are non-monotonic by definition](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), and [the StatsD receiver sets counters as non-monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). Since StatsD is so widely used, when considering delta support, it's important to make sure non-monotonic counters will work properly in Prometheus. +[StatsD counters are non-monotonic by definition](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/1789), and [the StatsD receiver sets counters as non-monotonic by default](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md). Since StatsD is so widely used, when considering delta support, it's important to make sure non-monotonic counters will work properly in Prometheus. ### Pitfalls of the current solution @@ -72,7 +71,7 @@ As suggested in an [earlier delta doc](https://docs.google.com/document/d/1vMtFK #### No added value to conversion -Cumulative metrics are resilient to data loss - if a sample is dropped, the next sample will still include the count from the previous sample. With delta metrics, if a sample is dropped, its data is just lost. Converting from delta to cumulative doesn’t improve resiliency as the data is already lost before it becomes a cumulative metric. +Cumulative metrics are resilient to data loss - if a sample is dropped, the next sample will still include the count from the previous sample. With delta metrics, if a sample is dropped, its data is just lost. Converting from delta to cumulative doesn’t improve resiliency as the data is already lost before it becomes a cumulative metric. Cumulative metrics are usually converted into deltas during querying (this is part of what `rate()` and `increase()` do), so converting deltas to cumulative is wasteful if they’re going to be converted back into deltas on read. @@ -124,15 +123,15 @@ For the initial implementation, ignore `StartTimeUnixNano`. To ensure compatibil #### Set metric type as gauge -For this initial proposal, OTEL deltas will be treated as a special case of Prometheus gauges. When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. +For this initial proposal, OTEL deltas will be treated as a special case of Prometheus gauges. When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. -A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. +A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time. -Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over a specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. Another way of putting it is that deltas are already a subset of the Prometheus gauge type. +Prometheus already uses gauges to represent deltas. While gauges ingested into Prometheus via scraping represent sampled values rather than a count within a interval, there are other sources that can ingest "delta" gauges. For example, `increase()` outputs the delta count of a series over a specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time. Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types. Another way of putting it is that deltas are already a subset of the Prometheus gauge type. This approach leverages an existing Prometheus metric type, avoiding the changes to the core Prometheus data model. -As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. If the `--enable-feature=type-and-unit-labels` (from PROM-39) is set, a `__type__="gauge"` will be added as a label. +As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_types/), "The Prometheus server does not yet make use of the type information and flattens all data into untyped time series". Recently however, there has been [an accepted Prometheus proposal (PROM-39)](https://github.com/prometheus/proposals/pull/39) to add experimental support type and unit metadata as labels to series, allowing more persistent and structured storage of metadata than was previously available. This means there is potential to build features on top of the typing in the future. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label. If the `--enable-feature=type-and-unit-labels` (from PROM-39) is set, a `__type__="gauge"` will be added as a label. An alternative/potential future extension is to [ingest as Prometheus counters instead](#ingest-as-counter). That option is a bigger change since it alters the Prometheus model rather than reusing a pre-existing type. However, could be less confusing as it aligns the Prometheus data model more closely to the OTEL one. @@ -172,7 +171,7 @@ For the initial implementation, reuse existing chunk encodings. Delta counters will use the standard XOR chunks for float samples. -Delta histograms will use native histogram chunks with the `GaugeType` counter reset hint/header. The counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. A (bucket or total) count drop is detected as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Additionally, counter histogram chunks have the invariant that no count ever goes down baked into their implementation. `GaugeType` allows counts to go up and down, and does not cut new chunks on counter resets. +Delta histograms will use native histogram chunks with the `GaugeType` counter reset hint/header. The counter reset behaviour for cumulative native histograms is to cut a new chunk if a counter reset is detected. A (bucket or total) count drop is detected as a counter reset. As delta samples don’t build on top of each other, there could be many false counter resets detected and cause unnecessary chunks to be cut. Additionally, counter histogram chunks have the invariant that no count ever goes down baked into their implementation. `GaugeType` allows counts to go up and down, and does not cut new chunks on counter resets. ### Scraping @@ -180,9 +179,9 @@ No scraped metrics should have delta temporality as there is no additional benef ### Federation -Federating delta series directly could be usable if there is a constant and known collection interval for the delta series, and the metrics are scraped at least as often as the collection interval. However, this is not the case for all deltas and the scrape interval cannot be enforced. +Federating delta series directly could be usable if there is a constant and known collection interval for the delta series, and the metrics are scraped at least as often as the collection interval. However, this is not the case for all deltas and the scrape interval cannot be enforced. -Therefore we will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the delta labels are set. +Therefore we will add a warning to the delta documentation explaining the issue with federating delta metrics, and provide a scrape config for ignoring deltas if the delta labels are set. ### Remote write ingestion @@ -220,11 +219,11 @@ With `sum_over_time()`, the actual range covered by the sum could be different f * S1: StartTimeUnixNano: T0, TimeUnixNano: T2, Value: 5 * S2: StartTimeUnixNano: T2, TimeUnixNano: T4, Value: 1 -* S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 +* S3: StartTimeUnixNano: T4, TimeUnixNano: T6, Value: 9 -And `sum_over_time()` was executed between T1 and T5. +And `sum_over_time()` was executed between T1 and T5. -As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. +As the samples are written at TimeUnixNano, only S1 and S2 are inside the query range. The total (aka “increase”) of S1 and S2 would be 5 + 1 = 6. This is actually the increase between T0 (StartTimeUnixNano of S1) and T4 (TimeUnixNano of S2) rather than the increase between T1 and T5. In this case, the size of the requested range is the same as the actual range, but if the query was done between T1 and T4, the request and actual ranges would not match. **Example 2** @@ -244,8 +243,8 @@ Additionally, as mentioned before, it is common for deltas to have samples emitt To help users use the correct functions, warnings will be added if the metric type/temporality does not match the types that should be used with the function. The warnings will be based on the `__type__` label, so will only work if `--enable-feature=type-and-unit-labels` is enabled. Because of this, while not necessary, it will be recommended to enabled the type and unit labels feature alongside the delta support feature. -* Cumulative counter-specific functions (`rate()`, `increase()`, `irate()` and `resets()`) will warn if `__type__="gauge"`. -* `sum_over_time()` will warn if `type="counter"`. +* Cumulative counter-specific functions (`rate()`, `increase()`, `irate()` and `resets()`) will warn if `__type__="gauge"`. +* `sum_over_time()` will warn if `type="counter"`. ### Summary @@ -313,7 +312,7 @@ There is an effort towards adding CreatedTimestamp as a field for each sample ([ CT-per-sample is not a blocker for deltas - before this is ready, `StartTimeUnixNano` will just be ignored. -Having CT-per-sample can improve the `rate()` calculation - the collection interval for each sample will be directly available, rather than having to guess the interval based on gaps. It also means a single sample in the range can result in a result from `rate()` as the range will effectively have an additional point at `StartTimeUnixNano`. +Having CT-per-sample can improve the `rate()` calculation - the collection interval for each sample will be directly available, rather than having to guess the interval based on gaps. It also means a single sample in the range can result in a result from `rate()` as the range will effectively have an additional point at `StartTimeUnixNano`. There are unknowns over the performance and storage with this approach. @@ -323,7 +322,7 @@ For this initial proposal, the OTEL metric properties will only be added to delt ### Inject zeroes for StartTimeUnixNano -[CreatedAt timestamps can be injected as 0-valued samples](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection). Similar could be done for StartTimeUnixNano. +[CreatedAt timestamps can be injected as 0-valued samples](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection). Similar could be done for StartTimeUnixNano. CT-per-sample is a better solution overall as it links the start timestamp with the sample. It makes it easier to detect overlaps between delta samples (indicative of multiple producers sending samples for the same series), and help with more accurate rate calculations. @@ -333,7 +332,7 @@ It's possible for the StartTimeUnixNano of a sample to be the same as the TimeUn ### Rate calculation extensions (without CT-per-sample) -[Querying deltas](#querying-deltas) outlined the caveats of using `sum_over_time(...[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. +[Querying deltas](#querying-deltas) outlined the caveats of using `sum_over_time(...[]) / ` to calculate the increase for delta metrics. In this section, we explore possible alternative implementations for delta metrics. This section assumes knowledge of [Extended range selectors semantics proposal](https://github.com/prometheus/proposals/blob/main/proposals/2025-04-04_extended-range-selectors-semantics.md) which introduces the `smoothed` and `anchored` modifers to range selectors, in particular for `rate()` and `increase()` for cumulative counters. @@ -343,9 +342,9 @@ This also assumes CT-per-sample is not available. A possibly simpler solution if The reason why `increase()`/`rate()` need extrapolation for cumulative counters is to cover the entire range is that they’re constrained to only look at the samples within the range. This is a problem for both cumulative and delta metrics. -To work out the increase more accurately, the functions would also have to look at the sample before and the sample after the range to see if there are samples that partially overlap with the range - in that case the partial overlaps should be added to the increase. +To work out the increase more accurately, the functions would also have to look at the sample before and the sample after the range to see if there are samples that partially overlap with the range - in that case the partial overlaps should be added to the increase. -The `smoothed` modifer in the extended range selectors proposal does this for cumulative counters - looking at the points before and after the range to more accurately calculate the rate/increase. We could implement something similar with deltas, though we cannot naively use the propossed smoothed behaviour for deltas. +The `smoothed` modifer in the extended range selectors proposal does this for cumulative counters - looking at the points before and after the range to more accurately calculate the rate/increase. We could implement something similar with deltas, though we cannot naively use the propossed smoothed behaviour for deltas. The `smoothed` proposal works by injecting points at the edges of the range. For the start boundary, the injected point will have its value worked out by linearly interpolating between the closest point before the range start and the first point inside the range. @@ -361,9 +360,9 @@ Support for retaining `StartTimeUnixNano` would improve the calculation, as we w For cumulative counters, `increase()` works by subtracting the first sample from the last sample in the range, adjusting for counter resets, and then extrapolating to estimate the increase for the entire range. The extrapolation is required as the first and last samples are unlikely to perfectly align with the start and end of the range, and therefore just taking the difference between the two is likely to be an underestimation of the increase for the range. `rate()` divides the result of `increase()` by the range. This gives an estimate of the increase or rate of the selected range. -For consistency, we could emulate that for deltas. +For consistency, we could emulate that for deltas. -First sum all sample values in the range, with the first sample’s value only partially included if it's not completely within the query range. To estimate the proportion of the first sample within the range, assume its interval is the average interval betweens all samples within the range. If the last sample does not align with the end of the time range, the sum should be extrapolated to cover the range until the end boundary. +First sum all sample values in the range, with the first sample’s value only partially included if it's not completely within the query range. To estimate the proportion of the first sample within the range, assume its interval is the average interval betweens all samples within the range. If the last sample does not align with the end of the time range, the sum should be extrapolated to cover the range until the end boundary. The cumulative `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range. With sparse delta series, a long gap to a boundary is not very meaningful. The series could be ongoing but if there are no new increments to the metric then there could be a long gap between ingested samples. @@ -391,9 +390,9 @@ Function overloading could work on all metrics where `__type__="gauge"` label is Pros: -* Users would not need to know the temporality of their metric to write queries. Users often don’t know or may not be able to control the temporality of a metric (e.g. if they instrument the application, but the metric processing pipeline run by another team changes the temporality). +* Users would not need to know the temporality of their metric to write queries. Users often don’t know or may not be able to control the temporality of a metric (e.g. if they instrument the application, but the metric processing pipeline run by another team changes the temporality). * This is helpful when there are different sources which ingest metrics with different temporalities, as a single function can be used for all cases. -* Greater query portability and reusability. Published generic dashboards and alert rules (e.g. [Prometheus monitoring mixins](https://monitoring.mixins.dev/)) can be reused for metrics of any temporality, reducing operational overhead. +* Greater query portability and reusability. Published generic dashboards and alert rules (e.g. [Prometheus monitoring mixins](https://monitoring.mixins.dev/)) can be reused for metrics of any temporality, reducing operational overhead. Cons: @@ -402,7 +401,7 @@ Cons: * This gets further complicated when considering how functions could be combined. For example, a user may use recording rules to compute the sum of a delta metric over a shorter period of time, so they can later for the sum over a longer period of time faster by using the pre-computed results. `increase()` and `sum_over_time()` would both work for the recording rule. For the longer query, `increase()` will not work - it would return a warning (as the pre-computed values would be gauges/untyped and should not be used with `increase()`) and a wrong value (if the cumulative counter logic is applied when there is no `__temporality__="delta"` label). `sum_over_time()` for the longer query would work in the expected manner, however. * Migration between delta and cumulative temporality for a series may seem seamless at first glance - there is no need to change the functions used. However, the `__temporality__` label would mean that there would be two separate series, one delta and one cumulative. If you have a long query (e.g. `increase(...[30d]))`, the transition point between the two series will be included for a long time in queries. Assuming the [proposed metadata labels behaviour](https://github.com/prometheus/proposals/blob/main/proposals/0039-metadata-labels.md#milestone-1-implement-a-feature-flag-for-type-and-unit-labels), where metadata labels are dropped after `rate()` or `increase()` is applied, two series with the same labelset will be returned (with an info annotation about the query containing mixed types). * One possible extension could be to stitch the cumulative and delta series together and return a single result. -* There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. +* There is currently no way to correct the metadata labels for a stored series during query time. While there is the `label_replace()` function, that only works on instant vectors, not range vectors which are required by `rate()` and `increase()`. If `rate()` has different behaviour depending on a label, there is no way to get it to switch to the other behaviour if you've accidentally used the wrong label during ingestion. Open questions and considerations: @@ -414,11 +413,11 @@ Open questions and considerations: If we were to implement function overloading for `rate()` and `increase()`, how exactly will it behave for deltas? A few possible ways to do rate calculation have been outlined, each with their own pros and cons. -Also to take into account are the new `smoothed` and `anchored` modifiers in the extended range selectors proposal. +Also to take into account are the new `smoothed` and `anchored` modifiers in the extended range selectors proposal. A possible proposal would be: -* no modifier - just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). +* no modifier - just use use `sum_over_time()` to calculate the increase (and divide by range to get rate). * `anchored` - same as no modifer. In the extended range selectors proposal, anchored will add the sample before the start of the range as a sample at the range start boundary before doing the usual rate calculation. Similar to the `smoothed` case, while this works for cumulative metrics, it does not work for deltas (if CT-per-sample is not implemented). To get the same output in the cumulative and delta cases given the same input to the initial instrumented counter, the delta case should use `sum_over_time()`. * `smoothed` - Logic as described in [Lookahead and lookbehind](#lookahead-and-lookbehind-of-range). @@ -426,17 +425,17 @@ One problem with reusing the range selector modifiers is that they are more gene ### `delta_*` functions -An alternative to function overloading, but allowing more choices on how rate calculation can be done would be to introduce `delta_*` functions like `delta_rate()` and having range selector modifiers. +An alternative to function overloading, but allowing more choices on how rate calculation can be done would be to introduce `delta_*` functions like `delta_rate()` and having range selector modifiers. This has the problem of having to use different functions for delta and cumulative metrics (switching cost, possibly poor user experience). ### Treat as “mini-cumulative” -Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. +Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. This requires CT-per-sample (or some kind of precise CT tracking) to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. -The standard `rate()`/`increase()` functions do not work well with delta-specific characteristics, especially without CT-per-sample. The OTEL SDKs only emit datapoints when there is a change in the interval so samples might not come at consistent intervals. But `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate. +The standard `rate()`/`increase()` functions do not work well with delta-specific characteristics, especially without CT-per-sample. The OTEL SDKs only emit datapoints when there is a change in the interval so samples might not come at consistent intervals. But `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate. With CT-per-sample, `rate()` can make this information to more accurately calculate the intervals between samples. Assuming `rate()` continues to be constrained to only looking at the samples with timestamps within its bounds though, it will still not be able to accurately predict whether to extrapolate at the end of the time range is. The reason for extrapolating is that there might be a sample with timestamp outside of the range, but its CT is within the range of the query. With sparse delta series which only push updates when there is a change, a long gap between samples does not mean the series is definitively ending. @@ -456,7 +455,7 @@ Before committing to moving forward with any of the extensions , we should first ## Discarded alternatives -### Ingesting deltas alternatives +### Ingesting deltas alternatives #### Convert to rate on ingest @@ -472,11 +471,11 @@ Users might want to convert back to original values (e.g. to sum the original va This also does not work for samples missing StartTimeUnixNano. -#### Add delta `__type__` label values +#### Add delta `__type__` label values Extend `__type__` from PROM-39 with additional delta types for any counter-like types (e.g. `delta_counter`, `delta_histogram`). The original types (e.g. `counter`) will indicate cumulative temporality. -This would prevent the problems of mixups with the `counter` type that [Ingest as counter with `__temporality__` label](#ingest-as-counter-with-__temporality__-label) has as it's a completely new type. +This would prevent the problems of mixups with the `counter` type that [Ingest as counter](#ingest-as-counter) has as it's a completely new type. An additional `__monotonicity__` property would need to be added as well. @@ -494,8 +493,8 @@ Have a convention for naming metrics e.g. appending `_delta_counter` to a metric #### Convert to cumulative on query -Delta to cumulative conversion at query time doesn’t have the same out of order issues as conversion at ingest. When a query is executed, it uses a fixed snapshot of data. The order the data was ingested does not matter, the cumulative values are correctly calculated by processing the samples in timestamp-order. +Delta to cumulative conversion at query time doesn’t have the same out of order issues as conversion at ingest. When a query is executed, it uses a fixed snapshot of data. The order the data was ingested does not matter, the cumulative values are correctly calculated by processing the samples in timestamp-order. No function modification is needed - all cumulative functions will work for samples ingested as deltas. -However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. \ No newline at end of file +However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions. From f603fbc21f0a91e261d8828b65fd94af9d7aad1f Mon Sep 17 00:00:00 2001 From: Fiona Liao Date: Fri, 12 Sep 2025 18:52:35 +0100 Subject: [PATCH 34/34] Add current status update (wait for CT) Signed-off-by: Fiona Liao --- proposals/0048-otel_delta_temporality_support.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0048-otel_delta_temporality_support.md b/proposals/0048-otel_delta_temporality_support.md index 76a6c0a..464cf79 100644 --- a/proposals/0048-otel_delta_temporality_support.md +++ b/proposals/0048-otel_delta_temporality_support.md @@ -1,3 +1,5 @@ +**In the delta support wg, we have changed our preferred approach from the one suggested in this proposal (type as gauges, add otel labels) and instead would like to jump straight to the [Treat as mini-cumulative](#treat-as-mini-cumulative) approach. This would allow us to support deltas within the existing framework and without requiring the rate function to do different things depending on type/temporality. This requires the [CT per sample proposal](https://github.com/prometheus/proposals/pull/60) to be implemented. This proposal will on hold until CT per sample has advanced enough for us to be confident it will work for deltas (or choose a different approach if it does not).** + # OTEL delta temporality support * **Owners:** @@ -433,7 +435,7 @@ This has the problem of having to use different functions for delta and cumulati Deltas can be thought of as cumulative counters that reset after every sample. So it is technically possible to ingest as cumulative and on querying just use the cumulative functions. -This requires CT-per-sample (or some kind of precise CT tracking) to be implemented. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. +This requires CT-per-sample (or some kind of precise CT tracking) to be implemented and it to be taken into acount for the `rate()`/`increase()` calculations. Just zero-injection of StartTimeUnixNano would not work all the time. If there are samples at consecutive intervals, the StartTimeUnixNano for a sample would be the same as the TimeUnixNano for the preceding sample and cannot be injected. The standard `rate()`/`increase()` functions do not work well with delta-specific characteristics, especially without CT-per-sample. The OTEL SDKs only emit datapoints when there is a change in the interval so samples might not come at consistent intervals. But `rate()` assumes samples in a range are equally spaced to figure out how much to extrapolate.