Asked to score the same response five times, a local LLM judge gave the identical answer 36% of the time. Asked to match a human rater's score, it matched only 36% of the time too. The judge is precise. Precision is not accuracy.

Key takeaways

A registered gate required Spearman correlation of at least 0.5 between the judge's scores and human helpfulness ratings. The actual figure was 0.395, significant but weak - the gate fired, and the claim that this judge tracks human judgement well enough to trust is rejected.

A second gate required the judge's own repeat-to-repeat variation to stay below 0.75 points on a 0-4 scale. The actual figure was 0.386 - the gate passed, with margin. The judge is internally consistent.

36% of items got the exact same score across five independent runs. Separately, only 36% of items had the judge's score match the human rater's at all.…