Add active alerts for alternator latencies

[GeoffMontee]

Please make sure that this is a feature request.

System information

• Scylla version (you are using): Any
• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

Scylla Monitoring provides great panels showing Alternator latencies, but it does not currently provide alerts when latencies deviate from the average.

Who will benefit with this feature?

Users who use Alternator and depend on low latencies.

Any Other info.

Here is my alert configuration:

-  name: AlternatorByOpLatencyAboveAverage
   description: alternator operation latency has deviated from average
   severity: 1
   trigger-after: 10m
   expression: group((avg_over_time(scylla_alternator_op_latency_bucket[10m])-avg_over_time(scylla_alternator_op_latency_bucket[1d]))/stddev_over_time(scylla_alternator_op_latency_bucket[1d]) > 25) by (cluster, op)
   team: support

[amnonh]

There are a few issues with the suggested alert, the main one being that looking at a bucket like that most likely doesn't do what you expect it to do.

I suggest using a P99 or P95 and setting some hardcoded limit, similar to what we do with CQL.

[GeoffMontee]

Hi @amnonh,

Thanks for the reply. I have some followup questions:

I inspected the "P95 Latencies" panel on the "Alternator" dashboard, and I see the following expression:

      "expr": "histogram_quantile(0.95, sum(rate(scylla_alternator_op_latency_bucket{instance=~\"[[node]]\",cluster=~\"$cluster|$^\", dc=~\"$dc\", shard=~\"[[shard]]|$^\", op=~\"$ops\"}[$__rate_interval])>0) by (op, le))",

Would a similar calculation be used in an alert like this? Or would there be a better way to do this?

[GeoffMontee]

Hi @wpaven @ruthea @ManjotS @pdbossman,

We should probably be specific about what we want to alert on. Something like this?:

Alert if the p95 latency for BatchGetItem is greater than 8 ms

Or would someone define it differently?

Or do we want to define alerts for each operation, not just BatchGetItem?

[ManjotS]

I would rather look for patterns because it will differ per customer and cluster.

Here is my latest iteration

(histogram_quantile(0.95, sum(rate(scylla_alternator_op_latency_bucket{}[10m])>0) by (cluster, op, le))-histogram_quantile(0.95, sum(rate(scylla_alternator_op_latency_bucket{}[1d])>0) by (cluster, op, le)))/histogram_quantile(0.95, sum(rate(scylla_alternator_op_latency_bucket{}[1d])>0) by (cluster, op, le)) > 8

[ManjotS]

latest:

((avg(histogram_quantile(0.95, sum(rate(scylla_alternator_op_latency_bucket{}[10m])>0) by (cluster, op, le))) by (cluster))-(avg(histogram_quantile(0.95, sum(rate(scylla_alternator_op_latency_bucket{}[1d])>0) by (cluster, op, le))) by (cluster)))/(stddev(histogram_quantile(0.95, sum(rate(scylla_alternator_op_latency_bucket{}[1d])>0) by (cluster, op, le))) by (cluster)) > 2

[ManjotS]

We don't want this one widely distributed. This should be an SRE issue.

[ManjotS]

Please close this issue