Menu

Our statement job ran out of memory at seventy percent and Kubernetes started it again from the first customer
📰

Our statement job ran out of memory at seventy percent and Kubernetes started it again from the first customer

DEV Community: kubernetes·Sergey Shinder·15 days ago
#abWt6uqX
#devtokubernetes#devto#dev#statement#first
Reading 0:00
15s threshold

On the first working day of October about forty thousand of our customers received their monthly statement email twice, and a few thousand received it three times. The statement job had not been changed in a year. Our customer base had grown by a quarter since the spring.

The job runs once a month as a Kubernetes Job. It reads every account, renders a PDF, and calls our email provider for each customer in turn. Its pod had a memory limit of two gigabytes, which had been plenty when the account list was smaller. It also kept a summary of every statement sent in memory for a report at the end, and that grew with the customer count. In October it reached the limit about seventy percent of the way through. The kernel killed the container, the pod exited with code 137 and was marked failed.

A Job exists to reach a number of successful completions, and failure is something it is built to recover from.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.