Menu

Pub sub explained: topics, partitions, replicas, and failover
📰

Pub sub explained: topics, partitions, replicas, and failover

Reddit r/webdev·u/CorrectHornet4939·7 months ago
#SZut1o1P
#once#partitions#leader#topics#replicas
Reading 0:00
15s threshold

Most web apps start out with everything talking to the database directly, then hit a wall once you need background jobs, email sending, webhooks, analytics or anything async. That is usually when pub sub shows up .

Covers:

Producer, broker, consumer and how the three roles fit together

Inside the broker: a small control plane and a huge append only log

Why a log beats a queue: offsets, replay, fan out

Topics split into N partitions for throughput

Hash the key, modulo N, pick the partition, with a worked example

Replication with 3 replicas across 3 zones, one leader and two followers

Leader failover when the leader dies

Consumer groups and how partitions get split across consumers

At most once vs at least once vs exactly once delivery

The part that trips up most web devs moving from a job queue to something like Kafka is that the log model is fundamentally different. Messages are not deleted when consumed, consumers track their own offset , and you can replay history at will

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.