Menu

Sub-second OLAP without a warehouse: DuckDB + FastAPI
📰

Sub-second OLAP without a warehouse: DuckDB + FastAPI

DEV Community: fastapi·Aayush Kumar bhat·8 days ago
#rzdSoBjP
#devtofastapi#devto#dev#table#path
Reading 0:00
15s threshold

A dashboard does not need a warehouse on day one.

On the HVAC telemetry app I worked on, the analytical surface had to answer questions over a large PyArrow dataset while the product was still small. A warehouse would have meant a second system, a second deployment and a second bill. DuckDB gave us an in-process OLAP engine instead: the API was already Python, the data was already Parquet inside Delta Lake, and the queries needed to be fast, not distributed.

The public proof of concept for that app is here: poc_latency_test . It is a Next.js front end, a FastAPI service and a medallion pipeline. This is what the read path looks like.

The whole trick is not copying anything

DuckDB can query Parquet files directly. The pipeline already writes Delta tables, so the query path starts from the Delta table and hands DuckDB an Arrow dataset.

from deltalake import DeltaTable

def get_delta_dataset ( layer : str , table : str | None = None ): path = ( os . path .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.