Menu

1 / 2

Postmortem: How an Actix and Redis 8.0 Connection Leak Caused a 3-Hour Chat Outage

DEV Community·ANKUSH CHOUDHARY JOHAL·7 months ago
#cUYmWjdO
#postmortem#actix#redis#connection#pool
Reading 0:00
15s threshold

On November 14, 2024, at 09:42 UTC, our Rust-based chat service serving 1.2 million concurrent users dropped 92% of messages and hit 14-second p99 latency, traced to an Actix-Web 4.4 and Redis 8.0 connection leak that took 3 hours to resolve, costing $47k in SLA penalties and user churn.

📡 Hacker News Top Stories Right Now

Ghostty is leaving GitHub (452 points)

OpenAI models coming to Amazon Bedrock: Interview with OpenAI and AWS CEOs (50 points)

A playable DOOM MCP app (51 points)

Waymo in Portland (154 points)

Your phone is about to stop being yours (549 points)

Key Insights

Unbounded Redis connection pools in Actix-Web 4.4 with Redis 8.0's new connection lifecycle defaults caused 12,400 leaked connections in 47 minutes of peak load.

Actix-Web 4.4's default connection pool integration with redis-rs 0.23.1 lacks built-in idle timeout enforcement, a gap exacerbated by Redis 8.0's stricter maxclients default (10k vs 6.4k in 7.x).

The outage cost $47,200 in direct SLA penalties, plus an estimated…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.