Menu

Our Redis replica spent three days restarting the same sync
📰

Our Redis replica spent three days restarting the same sync

DEV Community: redis·Sergey Shinder·18 days ago
#RYwy8132
#devtoredis#devto#dev#replica#primary
Long article (391 words) — no AI summary yet.
Reading 0:00
15s threshold

In October we patched the host running the primary of our delivery tracking Redis, expecting Sentinel to promote the replica as it always had. It did not. The driver app could not sync for six minutes while the host rebooted, and when we looked at the replica it had not held a current copy of the data for three days.

Three days earlier there had been a forty second network interruption during switch maintenance. When a replica reconnects, the primary tries a partial resync from its replication backlog, a ring buffer of recent writes. Ours was at the default size of one megabyte. We write about fifteen megabytes a second at peak, so the backlog covered well under a second, and a forty second gap meant a full resync: the primary forks, produces a snapshot of the whole dataset and streams it to the replica.

While it does that, every new write for the replica is held in that replica's output buffer on the primary.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.