On November 14, 2024, at 09:42 UTC, our Rust-based chat service serving 1.2 million concurrent users dropped 92% of messages and hit 14-second p99 latency, traced to an Actix-Web 4.4 and Redis 8.0 connection leak that took 3 hours to resolve, costing $47k in SLA penalties and user churn.
📡 Hacker News Top Stories Right Now
Ghostty is leaving GitHub (452 points)
OpenAI models coming to Amazon Bedrock: Interview with OpenAI and AWS CEOs (50 points)
A playable DOOM MCP app (51 points)
Waymo in Portland (154 points)
Your phone is about to stop being yours (549 points)
Key Insights
Unbounded Redis connection pools in Actix-Web 4.4 with Redis 8.0's new connection lifecycle defaults caused 12,400 leaked connections in 47 minutes of peak load.
Actix-Web 4.4's default connection pool integration with redis-rs 0.23.1 lacks built-in idle timeout enforcement, a gap exacerbated by Redis 8.0's stricter maxclients default (10k vs 6.4k in 7.x).
The outage cost $47,200 in direct SLA penalties, plus an estimated…