You cordon a node, run kubectl drain , and the terminal prints the same line every five seconds:

error when evicting pods/"api-7d9c6b8f5-x2k4q" -n payments (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Enter fullscreen mode

Exit fullscreen mode

Nothing is broken. Drain is doing what it was told: asking the API server to evict pods one at a time and waiting. A drain hangs for a small number of reasons, and almost all of them can be read from the cluster before you start. This post goes through them with commands you can run against any cluster.

What drain actually does

kubectl drain <node> first cordons the node (marks it unschedulable), then evicts each pod through the Eviction API. Eviction is not delete. The API server checks the pod's PodDisruptionBudget (PDB) and refuses with HTTP 429 if the eviction would take the workload below its budget.…