Hey all 👋

Last episode was an RCA with seven root causes and a port number. This one is an RCA with one root cause, five wrong theories, and a cluster that reported itself healthy for thirty-eight hours while doing absolutely nothing.

State of the lab going in: two nodes, Flux in charge, Airflow on a public domain through a Cloudflare tunnel, nightly backups, DHCP reservations finally ticked. Episode 10 ended with "the router remains the weakest member of the team." I would like to amend that statement. The router has competition.

Here's what happened, in the order I found out, which is not the order it happened.

🚨 Symptom: The Tunnel Is Down

Sunday night, 10 PM. Cloudflare's Zero Trust dashboard, which has shown two happy connectors since Episode 9, says the tunnel is down. No connectors. Nothing.

First look at the cluster:

$ kubectl get nodes NAME STATUS ROLES AGE VERSION dell NotReady <none> 64d v1.35.6 msi Ready <none> 64d v1.35.6

Enter fullscreen mode

Exit fullscreen mode

Dell is down.…