This is part of my continuing take on developing sophisticated optimization frameworks for agents. My previous article addressed head to head configuration comparison to identify superior combinations of model, tools and prompts. In this article I’ll address merging competing eval metrics into a single, meaningful score that captures multi-team objectives. Used together, teams can identify which pieces to optimize without regressions.

The business problem comes first

A recent survey has addressed the different priorities that are pulling on production agent systems. It has to be accurate and provide a high quality output to users. It needs to be on budget. It needs to run quickly. There is a heuristic about hiring trades contractors: you can be on time, on budget or on quality… pick two out of three. It is very difficult to deliver all three at the highest standards, but you start by setting minimum standards and constantly pushing to be better.

The cost of ignoring this is measurable.…