KubeletDown #
Meaning #
This alert is triggered when the monitoring system has not been able to reach any of the cluster’s Kubelets for more than 15 minutes.
Impact #
This alert represents a critical threat to the cluster’s stability. Excluding
the possibility of a network issue preventing the monitoring system from
scraping Kubelet metrics, multiple nodes in the cluster are likely unable to
respond to configuration changes for pods and other resources, and some
debugging tools are likely not functional, e.g. kubectl exec and kubectl logs.
Diagnosis #
Check the status of nodes and for recent events on Node objects, or for recent
events in general:
$ kubectl get nodes
$ kubectl describe node $NODE_NAME
$ kubectl get events --field-selector 'involvedObject.kind=Node'
$ kubectl get events
If you have SSH access to the nodes, access the logs for the Kubelet directly:
$ journalctl -b -f -u kubelet.service
Mitigation #
The mitigation depends on what is causing the Kubelets to become unresponsive. Check for wide-spread networking issues, or node level configuration issues.