-
Notifications
You must be signed in to change notification settings - Fork 1.9k
enhancement(aggregate): Add Distribution as an aggregation mode #23994
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
enhancement(aggregate): Add Distribution as an aggregation mode #23994
Conversation
…/jlambatl/vector into transform_aggregate_distribution
…improve the documentation that is generated for the distribution aggregation mode in the configuration parameter
|
I've pushed up a commit that should resolve the comments/feedback provided. It's hard to find the balance between being too verbose and not providing enough clarity. |
|
Starting to look good, will wait for a review from @vectordotdev/documentation. |
|
@pront The update looks good to me; it's also nice to see the improved tests. Thankyou. |
…transform_aggregate_distribution
Head branch was pushed to by a user without write access
|
@pront I've fixed the Clippy error that was failing in CI and pushed up the changes. |
Summary
This change adds a new AggregateMode
distributionthat aggregates individual metrics into a distribution. This doesn't reduce the accuracy of the metrics; instead, it reduces the volume of payloads being emitted from Vector by reducing the overhead of duplicate tags and other fields in exchange for a single larger payload.This pattern suits real-time analytics databases, where histograms and other statistical functions can be calculated on the fly. It reduces the need to manually codify histogram buckets, which can become an onerous task on systems dealing with large metric volumes.
The distribution is sorted in the current implementation. I have been unsure whether that is a good idea, but I'm happy to change that based on your feedback.
closes: #24001
Vector configuration
How did you test this PR?
I have added many different unit tests that cover the primary aggregation types, histograms, distributions, etc.
Change Type
Is this a breaking change?
Does this PR include user-facing changes?
no-changeloglabel to this PR.Notes
N/A