Every search team eventually ships a ranking change on the strength of a table where the new system's nDCG@10 is a little higher than the old one's. Then it goes to production and nothing measurable happens.
Usually nothing went wrong in the code. The evaluation was answering a different question from the one people thought they were asking. I built llm-search-eval to make the evaluation answer the right one: is the new system better for shoppers, and how sure are we? This post is about the design decisions behind it, in the order the problems show up.
All numbers below come from the small bilingual electronics example that ships in the repository (10 queries, 22 products, 30 gold labels). It is far too small to make product decisions with, which is exactly why it is useful for showing where a naive evaluation goes wrong.
Problem 1: labels run out, and the gap is not neutral
Offline evaluation needs relevance labels: for each (query, product) pair, how relevant is it? Teams label a set once and reuse it.…