Menu

Your eval set is the product. The prompt is just this week's implementation.
📰

Your eval set is the product. The prompt is just this week's implementation.

DEV Community: machinelearning·Onkar Deokate·about 2 hours ago
#4A5BAsJP
Reading 0:00
15s threshold

There's a new model almost every day this month. In October alone: GPT-6.1 Terra, Kimi K3.1 Pro, DeepSeek V4.1 Flash, MiniMax M3.1 Flash, Beam 1.0, Mercury 3, and more.

If your LLM feature is a prompt and a feeling, each of those releases is a question you can't answer: should we switch? Is it better for us ? Cheaper at the same quality? Worse in some way we'd only find out from users?

If your feature has an eval set, the question takes an afternoon.

What an eval set actually is

Not a benchmark. A list of your inputs with your definition of a correct answer, and a script that scores any model, prompt or design against it.

For a text-to-SQL feature I'm working on, each case is:

- question : " Which leads haven't been contacted in 30 days?" reference_sql : | SELECT l.id FROM leads l LEFT JOIN activities a ON a.lead_id = l.id GROUP BY l.id HAVING max(a.created_at) < now() - interval '30 days' OR max(a.created_at) IS NULL

Enter fullscreen mode

Exit fullscreen mode

The scorer runs both queries and…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.