AI agents have expanded inference from single-turn interactions into multi-step workflows that reason, invoke tools, coordinate subagents, and carry growing context from one turn to the next. The scale of this shift is now visible in raw consumption: across 100 trillion tokens of real-world usage, OpenRouter’s State of AI report found that average prompt tokens per request grew roughly fourfold, and single agentic requests consume 15 times the tokens of ordinary chat.

Properly characterizing hardware performance for this workload presents new challenges. A useful benchmark must capture long-context prefill, KV-cache reuse, interactive decode, tool-call gaps, and distributed mixture of experts (MoE) execution under realistic concurrency.…