When deploying large language models to production, measuring performance accurately is critical. Whether you're using vLLM, SGLang, TensorRT-LLM, or a custom inference stack, you need to understand:

Throughput : How many requests per second can your system handle?

Latency metrics : Time to First Token (TTFT), Inter-Token Latency (ITL), and end-to-end latency

Token generation speed : Tokens per second under different concurrency levels

Tail latency : P95 and P99 values that affect user experience

In this post, I'll walk through the key metrics for benchmarking language models and share why I built llmperf-rs , a Rust-based benchmarking tool that takes a different approach to measuring these metrics.

The Problem with Existing Tools

While working with ray-project/llmperf (now archived), I noticed that Inter-Token Latency (ITL) was calculated by averaging per-request first, then aggregating those averages.…