kubectl get pods -A takes eleven seconds. The web console spins. Controllers start logging client rate limiter Wait returned an error: context deadline exceeded . Someone restarts the API server pods, it gets slightly better for ten minutes, then it comes back.

Nine times out of ten this is not an API server problem. The API server is a stateless request translator; its p99 is almost entirely a function of how fast etcd can answer. And etcd's answer time is, in turn, almost entirely a function of two things: how long fsync takes on the underlying disk, and how much data has to be walked through to serve a range request. This article is about measuring both precisely, then fixing them.

Why etcd latency becomes everyone's problem

etcd is a Raft-replicated, write-ahead-logged key-value store.…