Menu

Lessons learned benchmarking eBPF (Grafana Beyla) vs. OTel + Linux PSI + cAdvisor for diagnosing LLM tail-latency spikes on Kubernetes

#kubernetes#redditkubernetes#pods#false#json
Reading 0:00
15s threshold

While investigating why quantized `llama.cpp` inference pods behind a FastAPI gateway on a 3-node `kind` cluster were experiencing sudden P95 latency spikes (500ms jumping to 30s) and HTTP 502/504 timeouts, I ran a controlled benchmark across 64 reproducible fault and decoy runs to see which signals actually isolate the root cause.

Here are the 4 most interesting findings from the experiments:

**CFS CPU Throttling vs. Host CPU Contention (Linux PSI):**

Looking only at pod CPU utilization misses the distinction between a container hitting its own `limits.cpu` (`container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total > 90%`) versus a noisy neighbor saturating the worker node's runqueue (`container_pressure_cpu_waiting_seconds_total` Linux PSI spike with 0% local CFS throttling). The remediation is completely different (raising CPU limits vs.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.