Part 3 of the LayerCache series.
Part 1 — why I built it.
Part 2 — how to use it.
I want to talk about the incidents.
Not the elegant architecture diagrams. Not the benchmark numbers. The 2am Slack alerts. The postmortems. The moments where I stared at a graph thinking "the cache is supposed to prevent this."
Every incident in this post is real. And each one taught me something that I eventually baked into LayerCache .
Incident #1: The Database Died at 9:02 AM on a Monday
The alert: DB CPU at 100%. Query queue backing up. Response times climbing past 30 seconds.
The timeline:
9:00 AM — regular Monday morning traffic ramp-up begins
9:01 AM — a popular product listing key expires (it was set with a fixed TTL over the weekend)
9:02 AM — 400+ users hit the product listing at the same moment
9:02 AM — all 400 requests miss the cache simultaneously and slam the DB
Classic cache stampede.…