Rabata is S3-compatible object storage running on our own hardware in the EU and the US. In this four-part series our engineers explain how the stack works under the hood. Part 1 covered the overall architecture. This part is about s3-front, the service that holds all S3 metadata.

Why Redis

All S3 metadata lives in Redis. So does metadata in our other services. The main argument is speed: request latency is measured in microseconds, not in the milliseconds of relational databases. Object metadata is small, uniform, and read far more often than written.

Redis also comes with constraints. Everything lives in memory, so data volume is bounded by RAM. Redis is single-threaded, so it scales through sharding and replicas. Three more constraints define our whole data model.

No transactions: one script per S3 operation

Redis has no transactions, or more precisely, no atomicity across commands.…