While investigating why quantized `llama.cpp` inference pods behind a FastAPI gateway on a 3-node `kind` cluster were experiencing sudden P95 latency spikes (500ms jumping to 30s) and HTTP 502/504 timeouts, I ran a controlled benchmark across 64 reproducible fault and decoy runs to see which signals actually isolate the root cause.
Here are the 4 most interesting findings from the experiments:
**CFS CPU Throttling vs. Host CPU Contention (Linux PSI):**
Looking only at pod CPU utilization misses the distinction between a container hitting its own `limits.cpu` (`container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total > 90%`) versus a noisy neighbor saturating the worker node's runqueue (`container_pressure_cpu_waiting_seconds_total` Linux PSI spike with 0% local CFS throttling). The remediation is completely different (raising CPU limits vs.…