On October 12, 2024, a misconfigured Kafka 3.8 broker idempotent write setting caused a 20-minute total outage that delayed 1,047,892 order messages, cost $42k in SLA penalties, and exposed a critical gap in our rolling upgrade validation pipeline.

📡 Hacker News Top Stories Right Now

LLMs consistently pick resumes they generate over ones by humans or other models (34 points)

How fast is a macOS VM, and how small could it be? (144 points)

Why does it take so long to release black fan versions? (497 points)

Barman – Backup and Recovery Manager for PostgreSQL (31 points)

Refusal in Language Models Is Mediated by a Single Direction (21 points)

Key Insights

Kafka 3.8’s default idempotent producer retry backoff of 100ms combined with a 30-second broker failover window caused 89% of in-flight messages to expire before redelivery

Kafka 3.8.1 (released 14 days post-outage) patches the default retry.backoff.ms to 500ms and adds upgrade pre-flight checks for idempotent write compatibility

Implementing the…