Menu

1 / 2

Calibrate the Task Before the Model: A Spec-Only Floor for Agent Benchmarks

DEV Community: ai·Casey Zhang·7 days ago
#DesYtm89
#devtoai#devto#dev#tasks#task
Reading 0:00
15s threshold

You publish a table. Model A sits at 71% on your patch suite, model B at 64%. Someone asks a simple question — how many of those tasks are solvable from the issue text alone? — and the thread goes quiet.

That silence is the bug. Seven points of separation means nothing if four of your tasks never needed the repository in the first place. Before you compare endpoints, you have to know how much of your dataset a prompt alone can already solve.

This post is a working method for that: a spec-only floor , computed per task, cheap enough to run on a free tier, and boring enough to publish alongside your scores.

The problem is task-side, not model-side

Most agent benchmarks are built by collecting real issues and real patches. That is good for realism and terrible for calibration. A task titled "rename getUserData to fetchUser " is a find-and-replace. A task titled "fix flaky integration test" is a research project.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.