This lab fixes the autoscaling gap CPU HPA misses: users are waiting, but CPU is barely moving.

You will reproduce the failure on an I/O-bound Python service, then use Splunk Observability, the OpenTelemetry Collector, and KEDA to scale on request latency instead. In my run, the app sat at 1 replica with 3049.2 ms peak latency and 2.8% peak CPU, then scaled from 1 to 5 ready replicas in 48 seconds once KEDA read the Splunk latency metric.

I work a lot with the OpenTelemetry Collector in Kubernetes, and I reproduce problems like this in a throwaway AWS account before I write about them.

Here is how it works, and how to build it yourself.

TL;DR

What you get: an EKS Deployment whose replica count follows request latency, not CPU.

The trick: the app exposes app_request_latency_ms , the Splunk OpenTelemetry Collector sends it to Splunk Observability Cloud, and KEDA feeds that value into an HPA.

The proof: before, replicas stayed at 1 ; after, they went 1 -> 5 under load and back to 1 after load…