Our stock availability service keeps a snapshot of every depot's stock in memory. In September we stopped building that snapshot inside the service at startup and moved the work to an init container, which downloads the latest snapshot from object storage, unpacks it and indexes it onto a shared emptyDir before the service starts. Unpacking is memory hungry, so the init container got a memory request of six gibibytes. The service itself requests five hundred and twelve mebibytes and has never used much more.
The week after, the cluster autoscaler added nine nodes, and our cost dashboard noticed before we did. The nodes were real and paid for. Their memory use was under fifteen percent.
The scheduler does not add an init container's request to the pod's. It takes the larger of two numbers: the highest request among the init containers, and the sum of the requests of the app containers. That larger number is the pod's effective request.…