Originally published at webofmike.com on 2026-11-02. The demo repo and every command in it were run before publishing.
Escalating every uncertain agent decision to a frontier model sounds like the safe default. On a 240-case benchmark it was also the worse one: routing by confidence and escalating only when a cheap model said it wasn't sure cost half as much as escalating everything, and scored higher.
jev-benchmark measured TypeSafe AI's Jev against Claude on the same agent tool-call risk task. jev-calibration went further and measured whether Jev's own confidence score could be trusted to make a routing decision, and found a specific, usable threshold: a confidence of exactly 1.000 was right 133 times out of 134 on 240 labelled cases. Neither post built the thing that threshold is for. "Escalate when confidence is low" is not a model feature, it is a two-call routing problem, and routing problems belong at a gateway.β¦