Prometheus alerts and missing data

Joe Gregorio

Alerting in Prometheus is great, and easy, but one of the gotchas is that there’s no warnings or errors if you write an alert rule and there’s no data for that alert.

Now Prometheus does have the absent() function, and you could tack

OR absent(some_metric_name) == 1

to the end of every alert you write, but that’s tedious and error prone. So I wrote promk-absent a quick little tool on Go to create a set of absence alerts based on an existing set of alerts. It just processes a single alerts YAML file and emits a new alerts YAML file with one absent alert for each alert in the original file. To install it run:

go get

Then run it with

promk-absent --input=your-rules.yml --output=absent-rules.yml

Don’t forget to include your new absent rules in your prometheus.yml file.

The tool does have the restriction that all expressions must be written in the form of:

expression relation constant

and so far that hasn’t been an issue for us.

comments powered by Disqus