Menu

1 / 8

GitHub - viggy28/streambed: Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol.

Hacker News·5 months ago
#LGbTvv2r
#github#postgres#streambed#query#iceberg
Reading 0:00
15s threshold

Postgres-to-Iceberg CDC engine. Offload analytical queries from your production database without changing your application.

streambed streams WAL changes via logical replication, writes Parquet files to S3, and commits Iceberg metadata. Query the result with any Iceberg-compatible engine -- or use the built-in query server, which speaks the Postgres wire protocol so you can connect with psql .

See It In Action

Same analytical query on pgbench (1M accounts, 500K history rows). Postgres on the left, Streambed on the right.

No ETL. No Spark. Just Postgres + S3.

Quick Start

<div dir="auto" data-snippet-clipboard-copy-content="# Start Postgres + MinIO locally docker compose up -d

# Build go build -o streambed ./cmd/streambed

# Start syncing + query server on :5433 ./streambed sync \ --source-url="postgres://postgres:test@localhost:5432/postgres" \ --s3-bucket="streambed" \ --s3-endpoint="http://localhost:9000" \ --s3-prefix="test" \ --query-addr=:5433

# Query your Postgres tables via Iceberg psql -h…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.