Hot partitions, shared reads, a live database migration, and the deleted messages that froze it at 99.9999%
One busy Discord channel could slow down conversations that had nothing to do with it.
Discord was storing trillions of messages across 177 database servers. Adding more servers didn't fix the real problem. So the engineers changed how requests reached the database, and then moved every message to a new one. When that move was 99.9999% complete, it stopped. The reason: messages people had already deleted.
This article follows a single message through all of it. If you'd rather watch, here's the animated version:
Watch Here
This is a historical case study. The migration finished in May 2022 and Discord described it in March 2023; the background comes from Discord's January 2017 post. It isn't Discord's architecture today. Diagrams of servers and traffic are illustrative.
Where a message lives
Start with what the app actually asks for. Open a channel and you need its newest messages.…