Introduction

Diagnosing tail-latency spikes in Kubernetes-based Large Language Model (LLM) inference workloads demands a precise methodology to differentiate between application-layer delays and resource contention. In a controlled experiment involving quantized llama.cpp inference pods deployed behind a FastAPI gateway on a 3-node kind cluster, we observed P95 latency spikes escalating from 500ms to 30s. To isolate root causes, we executed 64 reproducible fault injection runs, comparing eBPF-based observability tools like Grafana Beyla against a hybrid approach combining OpenTelemetry (OTel) spans, Linux Pressure Stall Information (PSI) metrics , and Kubernetes events .

The consequences of misdiagnosis are severe: incorrect remediations, such as unwarranted resource scaling or misconfigured CPU limits, can exacerbate performance degradation and diminish user experience.…