Skip to content

Expose metrics detailing differences between two alert sources

License

Notifications You must be signed in to change notification settings

weaveworks/alertdiff

Repository files navigation

Introduction

alertdiff is a service that continuously compares the alerts generated by a Prometheus-compatible implementation (e.g. Weave Cortex) with a reference Prometheus instance, and exports metrics which detail spurious or missing alerts.

Usage

  • Deploy alertdiff
  • Append e.g. http://alertdiff:8080/expected/api/v1/alerts to the --alertmanager.url of your reference Prometheus server
  • Add http://alertdiff:8080/actual/api/v1/alerts as an alertmanager to Cortex
  • Ensure both Prometheus & Cortex are configured with the same set of alerting rules and are receiving identical metrics

Then add some alert definitions based on the exported metrics:

ALERT AlertDiffMissingAlert
  IF alertdiff_missing_alert > 0
  FOR 1m

ALERT AlertDiffSpuriousAlert
  IF alertdiff_spurious_alert > 0
  FOR 1m

Getting Help

If you have any questions about, feedback for or problems with alertdiff:

Weaveworks follows the CNCF Code of Conduct. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting a Weaveworks project maintainer, or Alexis Richardson ([email protected]).

Your feedback is always welcome!

About

Expose metrics detailing differences between two alert sources

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published