Menu

1 / 2

Building a local-first decision engine that only calls the big model when it's worth it

DEV Community: machinelearning·Вячеслав Барсуков·6 days ago
#QTlsobUc
Reading 0:00
15s threshold

Every AI agent I've built eventually hits the same wall: calling a big model on every request is too expensive, but a small local model is too dumb on the hard cases.

Most teams pick one side and live with the consequences. I wanted a third option: run a small local model first, and escalate to a big one only when the expected benefit of escalation actually exceeds its cost.

That project is called decision-cascade . It's open source, it's on PyPI ( pip install decision-cascade ), and this post is about what I learned building it — including the parts that didn't work at first.

The problem with "escalate if confidence is low"

My first version did the obvious thing: if the local model's confidence < 0.6 , send the request to a bigger model. Simple, works, and completely wrong in practice.

Why wrong?…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.