There's a new model almost every day this month. In October alone: GPT-6.1 Terra, Kimi K3.1 Pro, DeepSeek V4.1 Flash, MiniMax M3.1 Flash, Beam 1.0, Mercury 3, and more.
If your LLM feature is a prompt and a feeling, each of those releases is a question you can't answer: should we switch? Is it better for us ? Cheaper at the same quality? Worse in some way we'd only find out from users?
If your feature has an eval set, the question takes an afternoon.
What an eval set actually is
Not a benchmark. A list of your inputs with your definition of a correct answer, and a script that scores any model, prompt or design against it.
For a text-to-SQL feature I'm working on, each case is:
- question : " Which leads haven't been contacted in 30 days?" reference_sql : | SELECT l.id FROM leads l LEFT JOIN activities a ON a.lead_id = l.id GROUP BY l.id HAVING max(a.created_at) < now() - interval '30 days' OR max(a.created_at) IS NULL
Enter fullscreen mode
Exit fullscreen mode
The scorer runs both queries and…