Why the Linux OOM killer killed your container when memory looked fine

Your phone buzzes at 3 AM. A pod restarted. kubectl describe pod says the last state was OOMKilled .

So you check the node. It has gigabytes free. Nothing is close to full. You stare at the two facts side by side: killed for lack of memory, and plenty of memory. One of them has to be lying.

Neither is. There are two different out-of-memory killers on that machine, and only one of them cares about the node's free memory.

The two killers

Every container on your node lives inside a cgroup , short for control group. A cgroup is the kernel's way of putting a fence around a group of processes and saying: this group gets this much CPU, this much memory, no more.

When you set resources.limits.memory: 512Mi on a container, you are setting the fence height for its cgroup. When the processes inside cross that line, the kernel steps in and kills something inside the cgroup. That is a cgroup OOM kill . The node can have 30 GB free.…