Menu

Same Prompt, Two Runs, 13 Points Apart: Why My .NET Eval Gate Needed a Third Answer
📰

Same Prompt, Two Runs, 13 Points Apart: Why My .NET Eval Gate Needed a Third Answer

DEV Community: dotnet·MICHAEL-MAURICE·about 11 hours ago
#dblCqGWx
#devtodotnet#devto#dev#runs#case
Reading 0:00
15s threshold

I ran the exact same prompt against the exact same 120 test cases, twice, with different seeds. Then I did it 30 times.

The pass rates differed by up to 13.3 points between two runs that changed nothing. My old CI rule, "block the build if the score drops more than 3 points below baseline", would have blocked 8 of those 30 builds. Eight red pipelines, zero real regressions.

That's the part of LLM evaluation nobody puts on the dashboard: one eval run is a sample, not a measurement. This post is about one narrow piece of the fix, the gate itself, and why "pass or fail" is the wrong shape for it.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, tested project behind this post (a .NET 10 eval service with Clean Architecture, judge calibration, paired gates and 215 passing tests, running offline against a simulator) plus a step-by-step PDF guide is inside.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.