Menu

1 / 2

CPU Humbled Me — A Kubernetes Throttling Story Hidden Between Prometheus Scrapes

DEV Community·Satyaki·6 months ago
#8yxDR1kw
#kubernetes#devops#sre#observability#limit
Reading 0:00
15s threshold

Memory is easy. CPU humbled me.

With memory, the rule is brutal but clear — cross the limit, the pod gets OOMKilled. Done.

CPU? CPU is sneaky. And I ignored it for the longest time… until it broke production.

Here's what happened 👇

We had an app running peacefully in-house. Then it went client-facing. Traffic surged, and suddenly ~15% of requests started timing out — most of them on DB calls.

I opened Grafana expecting a smoking gun. Nothing. CPU usage looked "fine." No throttling alerts screaming at me. Just confused timeouts.

The trap? Throttling happens in milliseconds. Prometheus scrapes every 15 seconds.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.