Menu

1 / 4

The Phantom OOM: How a Silent Session Leak Brought Down Our Microservice Every 48 Hours

Reading 0:00
15s threshold

For months, our secondary telemetry and notification microservice lived on a clock. Every 48 hours, like clockwork, Kubernetes would mercilessly terminate the pod with an OOMKilled (Out Of Memory) exit code.

At first, we treated it as a background tax of running high-throughput asynchronous Python applications. We simply scaled up the container memory limit from 512MB to 1GB, then to 2GB. Predictably, the crash cycle just stretched from 48 hours to 96 hours. The memory wasn't fluctuating based on load; it was climbing in a slow, relentless, linear line straight off a cliff.

If you’ve ever chased a phantom memory leak in a microservice architecture, you know how frustrating it is. Here is how we tracked down the culprit, hidden deep inside our ORM session lifecycle, how we diagnosed it, and the exact architectural fix that flattened our memory usage permanently.

1.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.