Our EDI poller collects the purchase orders that suppliers drop on our SFTP server and imports them into the ERP. It runs as a single pod with the Recreate strategy, because two pollers would download the same files. Our alert for pods that keep failing was the one most teams write first:
increase(kube_pod_container_status_restarts_total{namespace="integration"}[15m]) > 3
with for set to five minutes.
At ten to eleven on a Friday night in October a change went out with a new setting for the SFTP server's host key, written in the wrong format. The poller read it at startup and exited. The alert paged the engineer on call eight minutes later. Ten minutes after that, before he had his laptop open, it resolved. He looked at the dashboard, saw the pod running, and went back to bed. It crashed about every five minutes until twenty to nine on Monday, when purchasing asked why no supplier orders had arrived since Friday.…