I'm building evaluation adapters (small reward models) for Telegraph, a network that routes AI requests to miners and scores their answers. Disclosure: I'm competing in Telegraph Hackathon Season II, which has prizes.
A preference pair is a prompt, a better answer and a worse one. A reward model trained on pairs learns whatever separates the two answers. If something cheaper than the skill you want separates them, it learns that instead, and your accuracy number measures the shortcut.
The test
Before training, run rules that pick an answer without checking anything:
pick the longer answer
pick the answer with more words in common with the prompt (TF-IDF cosine)
pick the answer with more of an obvious feature (alarm words, a verdict word)
If any of them is far above 50%, fix the pairs first.
Two cases from this week
Another builder (AdversarialEval, in the Telegraph Discord) found a keyword-count judge got every one of their severity pairs right, because all the worse answers were downgrades.…