Watchdog #

Meaning #

This is an alert meant to ensure that the entire alerting pipeline is functional. This alert is always firing, therefore it should always be firing in Alertmanager and always fire against a receiver.

Impact #

If not firing then it should alert external systems that this alerting system is no longer working.

Diagnosis #

Misconfigured alertmanager, bad credentials, bad endpoint, firewalls.. Check alertmanager logs.

Mitigation #

There are integrations with various notification mechanisms that send a notification when this alert is not firing. For example the DeadMansSnitch integration in PagerDuty.