NodeFileDescriptorLimit #

Meaning #

This alert is triggered when a node’s kernel is found to be running out of available file descriptors – a warning level alert at greater than 70% usage and a critical level alert at greater than 90% usage.

Impact #

Applications on the node may no longer be able to open and operate on files. This is likely to have severe consequences for anything scheduled on this node.

Diagnosis #

You can open a shell on the node and use the standard Linux utilities to diagnose the issue:

$ NODE_NAME='<value of instance label from alert>'

$ oc debug "node/$NODE_NAME"
# sysctl -a | grep 'fs.file-'
fs.file-max = 1597016
fs.file-nr = 7104       0       1597016
# lsof -n

Mitigation #

Reduce the number of files opened simultaneously by either adjusting application configuration or by moving some applications to other nodes.