At 14:37 UTC on October 12, 2024, our production Redis 8 cluster hit 100GB of used memory, crashing three downstream microservices and triggering a SEV-1 incident that cost us $42k in SLA penalties and engineering hours.

📡 Hacker News Top Stories Right Now

Valve releases Steam Controller CAD files under Creative Commons license (1231 points)

Diskless Linux boot using ZFS, iSCSI and PXE (43 points)

Appearing productive in the workplace (894 points)

Permacomputing Principles (73 points)

SQLite Is a Library of Congress Recommended Storage Format (120 points)

Key Insights

Redis 8’s lazy-expire mechanism only evicts keys on access or during periodic background scans, leaving 100GB of orphaned keys uncollected for 72 hours

Redis 8.0.2’s default maxmemory-policy is noeviction, which causes write failures instead of evicting keys when memory is full

Adding TTLs to 12M keys reduced our monthly Redis infrastructure cost from $5.8k to $1.2k, a 79% reduction

Redis 9 will introduce mandatory TTL warnings for…