Alerting in Prometheus is great, and easy, but one of the gotchas is that there’s no warnings or errors if you write an alert rule and there’s no data for that alert. Now Prometheus does have the absent() function, and you could tack OR absent(some_metric_name) == 1 to the end of every alert you write, but that’s tedious and error prone. So I wrote promk-absent a quick little tool on Go to create a set of absence alerts based on an existing set of alerts.

We just finished migrating all of our monitoring from InfluxDB to Prometheus and I thought I'd write up our reasons for the change. Please note that these are my own personal observations and relate to a specific project, these issue may not apply to you and you should evaluate each product for your own uses. Update: To clarify, the versions of InfluxDB and Prometheus that I am talking about are InfluxDB 1.