Menu

Troubleshoot etcd Clusters Easily with Ivory
📰

Troubleshoot etcd Clusters Easily with Ivory

DEV Community: database·Andrei Sergeev·16 days ago
#4a7j50Oz
#devtodatabase#devto#dev#etcd#member
Reading 0:00
15s threshold

See the leader, the Raft lag and the alarms of every etcd member in one screen — and move the leader with one click

etcd is one of those databases you don't think about. It sits quietly under your Patroni cluster, or under your Kubernetes control plane, and keeps everyone agreeing on who the leader is.

Until it doesn't. When etcd hurts, everything built on top of it hurts too. Patroni can't renew its leader lock, the Postgres primary demotes itself, and suddenly your "database incident" is actually an etcd incident.

I added etcd support to Ivory for exactly this reason. Patroni uses etcd as its coordination store, so if Ivory looks after Patroni, it should look after etcd too.

The problems

Here's what troubleshooting etcd usually looks like for me:

etcdctl needs a lot of flags. Endpoints, certificates, --write-out=table , --cluster … Every time, I have to remember the exact incantation, and type it on the right host.

"Who is the leader?" is a three-endpoint question.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.