Most engineers reach for Spark or BigQuery the moment they hear "10 million records."

I did too — until I tried DuckDB.

What happened next surprised me: 47 seconds, on my laptop, with 4GB RAM.

No cluster. No cloud bill. No YAML configuration files.

Let me show you exactly how I did it.

🤔 Why DuckDB?

DuckDB is an in-process analytical database — think SQLite, but built for OLAP workloads.

It runs entirely in memory using columnar storage and vectorized execution.

The numbers speak for themselves:

Tool 10M Records Query Time Infrastructure

Pandas ~4.2 minutes Local

PySpark ~1.8 minutes Local cluster setup

DuckDB 47 seconds Local (no setup)

Polars ~55 seconds Local

🛠️ Setup (30 seconds)

pip install duckdb pandas

Enter fullscreen mode

Exit fullscreen mode

That's it. No Docker. No JVM.…