Menu

My Cache Caused 4 Production Incidents. Here's What I Learned.

DEV Community: redis·날다람쥐·7 months ago
#jGdlQbZE
#dev#class#strong#code#highlight
Reading 0:00
15s threshold

Part 3 of the LayerCache series.

Part 1 — why I built it.

Part 2 — how to use it.

I want to talk about the incidents.

Not the elegant architecture diagrams. Not the benchmark numbers. The 2am Slack alerts. The postmortems. The moments where I stared at a graph thinking "the cache is supposed to prevent this."

Every incident in this post is real. And each one taught me something that I eventually baked into LayerCache .

Incident #1: The Database Died at 9:02 AM on a Monday

The alert: DB CPU at 100%. Query queue backing up. Response times climbing past 30 seconds.

The timeline:

9:00 AM — regular Monday morning traffic ramp-up begins

9:01 AM — a popular product listing key expires (it was set with a fixed TTL over the weekend)

9:02 AM — 400+ users hit the product listing at the same moment

9:02 AM — all 400 requests miss the cache simultaneously and slam the DB

Classic cache stampede.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.