This is article 3 of 10 in Beyond Kubernetes Defaults . Read article 2: Kubernetes Has Garbage Collection. Why Did We Build Another One? first, or start with article 1: When Kubernetes Primitives Aren't Enough . Later articles cover custody, admission and more.

Kubernetes makes leader election straightforward.

With controller-runtime or the Lease API, replicas compete for a Lease and one becomes leader. When that leader stops renewing, another replica can take over.

For many controllers, that is sufficient.

For systems where leader-only actions mutate durable authority, we found a second question matters just as much:

How do we prove that the old leader can no longer act?

That is a fencing problem.

Election and fencing are different properties

Suppose replica A holds leadership.

A = leader B = follower

Enter fullscreen mode

Exit fullscreen mode

A pauses during a network or runtime failure.…