Menu

Our restart alert resolved itself after ten minutes and the pod kept crashing until Monday
📰

Our restart alert resolved itself after ten minutes and the pod kept crashing until Monday

DEV Community: kubernetes·Sergey Shinder·12 days ago
#jJ36DMjH
#devtokubernetes#devto#dev#minutes#alert
Reading 0:00
15s threshold

Our EDI poller collects the purchase orders that suppliers drop on our SFTP server and imports them into the ERP. It runs as a single pod with the Recreate strategy, because two pollers would download the same files. Our alert for pods that keep failing was the one most teams write first:

increase(kube_pod_container_status_restarts_total{namespace="integration"}[15m]) > 3

with for set to five minutes.

At ten to eleven on a Friday night in October a change went out with a new setting for the SFTP server's host key, written in the wrong format. The poller read it at startup and exited. The alert paged the engineer on call eight minutes later. Ten minutes after that, before he had his laptop open, it resolved. He looked at the dashboard, saw the pod running, and went back to bed. It crashed about every five minutes until twenty to nine on Monday, when purchasing asked why no supplier orders had arrived since Friday.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.