Skip to content

Aggregated histogram, distribution and timing values are all flushed with the first call's cardinality #999

Open

Description

@sebxt

What happens

With client side aggregation enabled and max_metric_samples_per_context set, histogram(), distribution() and timing() calls that share a name and tags but pass different cardinality values are stored in one context. On flush every sample is sent with the cardinality of whichever call created the context first.

#980 fixed the same problem for count, gauge and set by including the cardinality in the aggregation context key, but Aggregator.add_max_sample_metric still calls get_context(name, tags) without it.

What should happen

Each value should be flushed with the cardinality it was submitted with, as count, gauge and set already do.

Minimal repro

from datadog.dogstatsd import DogStatsd

sent = []
statsd = DogStatsd(disable_telemetry=True, disable_buffering=True,
                   disable_background_sender=True, disable_aggregation=False,
                   max_metric_samples_per_context=10)
statsd._send = lambda payload, replay_safe=False: sent.append(payload)
statsd.histogram("request.size", 1, cardinality="low")
statsd.histogram("request.size", 2, cardinality="high")
statsd.flush_aggregated_metrics()
print(sent)

Output:

['request.size:1|h|card:low', 'request.size:2|h|card:low']

Expected:

['request.size:1|h|card:low', 'request.size:2|h|card:high']

Root cause

Aggregator.add_max_sample_metric in datadog/dogstatsd/aggregator.py builds the context key with self.get_context(name, tags), leaving out the cardinality. MaxSampleMetricContexts.sample creates the MaxSampleMetric with the first call's cardinality and reuses it for later calls with the same key, so the later cardinalities are dropped.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions