I keep thinking we have finally covered all the ways we can get into trouble with our database. But then something completely unexpected goes wrong, and I learn about another way a database can surprise us.

At Photoroom, our backend handles roughly 2,500 requests per second, with around 10 TB of data. We run zero-downtime deployments and have availability SLOs of four or five nines, depending on the service. Yet some of our most surprising database problems have started with an ordinary change meeting a production condition we had not accounted for.

We now deploy hundreds of times a week, often with code written by agents and no human reviewing the change. We need systems that catch the mistakes we already understand, and enough visibility to recognise the ones we don’t. Here are six lessons that have helped us do that.

1. Batch and throttle backfills

During a large backfill, a lagging read replica threatened the primary database’s disk capacity.…