Menu

1 / 7

Postgres to DuckDB with CDC: One ATTACH Instead of a Nightly Export

DEV Community: postgres·Shubham Baldava·1 day ago
#m8GASyN8
#devtopostgresql#devto#dev#duckdb#catalog
Reading 0:00
15s threshold

DuckDB makes analytics on a laptop feel free, right up to the point where the data has to come from somewhere. For a lot of people that somewhere is Postgres. The first Postgres to DuckDB setup is usually a pg_dump or a CSV export loaded into a DuckDB file, and that file is stale by the time anyone opens it.

The usual ways to put Postgres data in front of DuckDB:

The DuckDB postgres extension . DuckDB logs in to the live database and reads the current rows. Nothing is copied and there is nothing to operate, but every query lands on Postgres.

pg_dump or COPY TO , then COPY into DuckDB. A script and a scheduler rebuild a DuckDB file. Freshness is hours to daily, and it does full reloads only, so updates and deletes never arrive as changes.

Debezium and Kafka, with a consumer that writes Parquet. Changes show up within seconds.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.