Menu

You went to bathroom.... and got crashed
📰

You went to bathroom.... and got crashed

DEV Community: redis·Bhanu partap·17 days ago
#Y5IJFxUo
#devtoredis#devto#dev#distributedsystems#redis
Reading 0:00
15s threshold

Imagine you are in a bathroom with a 5 minute timer.

You go in but crash at minute 3. The lock is still on for 2 more minutes.

Your friend arrives at minute 3 and just stands outside waiting even though the bathroom is empty.

That wasted time is exactly what happens in software when a process dies but the Redis lock stays alive.

Now imagine three servers behind a load balancer. Every 5 minutes a cron job fires to generate a daily report and all three try to grab it. You use Redis SETNX with a 5 minute TTL. Instance A wins and thinks it owns the bathroom. But then it gets OOM killed at minute 3. The lock remains. No report runs for 2 more minutes. Then another instance grabs it. But wait, the first instance restarted and also thinks it still owns the bathroom. Now both are writing to the same output. Data corruption. 💥

This is the split brain problem.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.