Originally published at woitzik.dev
Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.
My Prometheus ran for months with no resource requests or limits. Nobody noticed until it landed on vm-srv-k3s-13 , at the time the smallest node in the cluster with 4GB, and used about 1.4Gi there. Together with everything else scheduled on that node it pushed memory to 100%, and containerd kill timeouts started restarting unrelated pods. Three weeks later the TSDB was filling its NFS volume. This post covers the values that fixed both, what the live numbers look like now, and one restart problem that looked like memory but wasn't.β¦