TL;DR: The thing that broke my comfortable ignorance about storage engines was a pipeline ingesting sensor telemetry — about 50,000 inserts per second into a PostgreSQL 15 cluster. The hardware wasn't cheap: NVMe drives, 32 cores, 128GB RAM.

📖 Reading time: ~42 min

What's in this article

The Problem That Made Me Actually Care About Storage Engines

What an LSM Tree Actually Does (Without the Textbook Nonsense)

The Write Path Step by Step

The Read Path — Why It's More Expensive Than You Think

Compaction: The Thing That Keeps LSM Trees From Falling Apart

Setting Up RocksDB and Hitting the Real Rough Edges

How Cassandra and ScyllaDB Use LSM Differently Than RocksDB

LSM vs B-Tree Storage Engines: When You're Picking the Wrong Tool

The Problem That Made Me Actually Care About Storage Engines

The thing that broke my comfortable ignorance about storage engines was a pipeline ingesting sensor telemetry — about 50,000 inserts per second into a PostgreSQL 15 cluster.…