Skip to content

Fix infinit suffix add to counter metrics#3491

Closed
ehsannm wants to merge 1 commit into
open-telemetry:mainfrom
ehsannm:main
Closed

Fix infinit suffix add to counter metrics#3491
ehsannm wants to merge 1 commit into
open-telemetry:mainfrom
ehsannm:main

Conversation

@ehsannm
Copy link
Copy Markdown

@ehsannm ehsannm commented Nov 23, 2022

No description provided.

@ehsannm
Copy link
Copy Markdown
Author

ehsannm commented Nov 23, 2022

There is critical bug in Prometheus exporter, counters are growing indefinitely by adding _total to the end of the metric.

Copy link
Copy Markdown
Contributor

@MrAlias MrAlias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will move our implementation out of compliance with the specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#sums

Monotonic Sum metric points MUST have _total added as a suffix to the metric name.

If someone already named their metric with a suffix of _total an additional _total suffix must be added still.

@ehsannm
Copy link
Copy Markdown
Author

ehsannm commented Nov 24, 2022

Thanks @MrAlias , but where should we check this infinit loop. The problem is with each new variableLabel change we get another _total added to the counter name.

HELP qlub_pos_requests_ratio_total

TYPE qlub_pos_requests_ratio_total counter

qlub_pos_requests_ratio_total{code="200",country_code="ae",method="get_order",vendor="dummy"} 6

HELP qlub_pos_requests_ratio_total_total

TYPE qlub_pos_requests_ratio_total_total counter

qlub_pos_requests_ratio_total_total{code="200",country_code="ae",method="get_table",vendor="dummy"} 1

@ehsannm
Copy link
Copy Markdown
Author

ehsannm commented Nov 24, 2022

ok i realized this bug is in current release tag v0.33.0 and is fixed in main branch. I close this PR. It should be fixed in next release.

@ehsannm ehsannm closed this Nov 24, 2022
@ehsannm
Copy link
Copy Markdown
Author

ehsannm commented Nov 24, 2022

BTW, i think it is critical bug, why not release a hotfix v0.33.1 for it ? @MrAlias

@jmacd
Copy link
Copy Markdown
Contributor

jmacd commented Dec 2, 2022

For the next reader, the fix went in here: #3369

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants