In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the "we're shipping v2" message typed.
Then I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%.
The judge, the answers, the rubric and temperature 0 were all the same. The only change was which answer sat in the slot labeled "A". That is LLM-as-judge position bias , and if your eval script calls judge(new, old) in a fixed order, your headline number is probably measuring it too.
TL;DR
LLM-as-judge position bias means a pairwise judge prefers whichever answer sits in a particular slot (usually the first) regardless of quality.
In my run, swapping A and B flipped 87 of 300 verdicts (29%) , and 78 of those flips went to whichever answer was in slot A.
The bias shows up in close calls. When two answers are near-equal, slot position acts as the tiebreaker.
Fix: judge every pair in both orders .…