Every AI agent I've built eventually hits the same wall: calling a big model on every request is too expensive, but a small local model is too dumb on the hard cases.
Most teams pick one side and live with the consequences. I wanted a third option: run a small local model first, and escalate to a big one only when the expected benefit of escalation actually exceeds its cost.
That project is called decision-cascade . It's open source, it's on PyPI ( pip install decision-cascade ), and this post is about what I learned building it — including the parts that didn't work at first.
The problem with "escalate if confidence is low"
My first version did the obvious thing: if the local model's confidence < 0.6 , send the request to a bigger model. Simple, works, and completely wrong in practice.
Why wrong?…