Manual polling is an anti-pattern; stream the truth directly from the source.
What We're Building
We are constructing a robust Change Data Capture (CDC) pipeline using Go. This system watches the Write-Ahead Log (WAL) of a PostgreSQL instance, captures row-level changes, transforms the payload into a standard domain model, and emits it to a downstream consumer.
The architecture relies on logical replication rather than physical log scanning to handle schema changes gracefully. The system must handle connection resets and maintain exactly-once semantics relative to the upstream store without duplicating processing work.
The high-level data flow looks like this:
+----------------+ +----------------+ +----------------+ | PostgreSQL |---->| CDC Reader |---->| Downstream | | WAL Stream | | (Go Service) | | System | +----------------+ +----------------+ +----------------+ ^ | ^ | v | +-----+------------+ | +----------+-------+ | Failover / | | | Offset Storage | | Offset Replay | | | (Kafka/Table) |β¦