Alertmanager Members Inconsistent

AlertmanagerMembersInconsistent #

Meaning #

At least one of alertmanager cluster members cannot be found.

Impact #

Diagnosis #

Check if IP addresses discovered by alertmanager cluster are the same ones as in alertmanager Service. Following example show possible inconsistency in Endpoint IP addresses:

$ kubectl describe svc alertmanager-main

Name:              alertmanager-main
Namespace:         monitoring

$ kubectl get pod -o wide | grep alertmanager-main

alertmanager-main-0                            5/5     Running   0          11d
alertmanager-main-1                            5/5     Running   0          2d16h     
alertmanager-main-2                            5/5     Running   0          6d  

Mitigation #

Deleting an incorrect Endpoint should trigger its recreation with a correct IP address.