Menu

One headless Service with zero Pods stopped every nginx router from starting (Monzo 2017), and why a normal Service wouldn't have

Reddit r/kubernetes·u/Interesting-Ease3774·3 days ago
#OFt8IdxB
#nginx#service#pods#router#resolves
Reading 0:00
15s threshold

A detail I hadn't fully appreciated until I rebuilt Monzo's 2017 outage: a normal Service always resolves (to its ClusterIP) even with no endpoints, but a headless Service ( clusterIP: None ) resolves straight to its ready Pods, so with zero Pods its name has no records at all.

Put that name in an nginx proxy_pass and nginx won't start: it resolves every upstream once at startup and exits if any lookup fails. Running routers don't notice, because they loaded the config before the route was added. Restart them all at once and everything behind the router goes down:

nginx: [emerg] host not found in upstream "cards-v2" in /etc/nginx/conf.d/default.conf:6

kubectl logs deploy/router --previous is the command that finds it, since the crash-looping container has usually restarted by the time you look.

Guide (free): https://learn.k8sight.in/go/nginx-upstream

Also new: CPU throttling from limits set to the average (Buffer), and Pods Running but not Ready after a dnsPolicy: None change (Salesforce).…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.