Menu

LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts
📰

LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

DEV Community: python·jidonglab·18 days ago
#wpJrKOWg
#devtopython#devto#dev#judge#answer
Reading 0:00
15s threshold

In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the "we're shipping v2" message typed.

Then I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%.

The judge, the answers, the rubric and temperature 0 were all the same. The only change was which answer sat in the slot labeled "A". That is LLM-as-judge position bias , and if your eval script calls judge(new, old) in a fixed order, your headline number is probably measuring it too.

TL;DR

LLM-as-judge position bias means a pairwise judge prefers whichever answer sits in a particular slot (usually the first) regardless of quality.

In my run, swapping A and B flipped 87 of 300 verdicts (29%) , and 78 of those flips went to whichever answer was in slot A.

The bias shows up in close calls. When two answers are near-equal, slot position acts as the tiebreaker.

Fix: judge every pair in both orders .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.