CDC with Debezium and Kafka: A Source-to-Sink Walkthrough
Change Data Capture (CDC) moves database changes as events instead of repeatedly querying whole tables. In this walkthrough, PostgreSQL is the source of truth, Debezium reads its write-ahead log (WAL), Kafka transports the change events, and a small Python consumer maintains a SQLite projection as the sink.
The example uses a customers table, but the same pattern applies to operational databases feeding search indexes, warehouses, caches, downstream services, or lakehouse ingestion pipelines.
The Architecture and Delivery Model
The pipeline has four responsibilities:
PostgreSQL β β logical decoding from WAL βΌ Debezium PostgreSQL connector β β Kafka Connect source task βΌ Kafka topic: shop.public.customers β β ordered per Kafka partition βΌ Python sink consumer β βΌ SQLite customer_projection
Enter fullscreen mode
Exit fullscreen mode
PostgreSQL records every committed change in its WAL.β¦