Menu

Don't Migrate Logs. Outlive Them.
📰

Don't Migrate Logs. Outlive Them.

DEV Community: elasticsearch·Elisha Sterngold·1 day ago
#4nGO9VjE
#devtoelasticsearch#devto#dev#write#cluster
Long article (1928 words) — no AI summary yet.
Reading 0:00
15s threshold

For two months, every log written into Shipbook was written twice.

Once into the old Elasticsearch cluster, the one we had been running for years. And once, in parallel, into a new one — a different Elasticsearch version, with a completely different way of organizing the data. Dual-write. The standard playbook for moving a database you cannot afford to lose: keep both copies, keep them in sync, prove the new one is right, then flip.

It worked. We are fully on the new cluster now; the old one is gone. But somewhere in the middle of those two months, a much smaller change — one I was forced to handle a completely different way, because dual-write was physically impossible for it — quietly showed me that I had built the entire rest of the migration the hard way.

That smaller change took one line of code and a calendar reminder. The big one took two months of machinery and a steady drip of bugs.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.