Menu

Shortcut judges: a cheap test your preference pairs should pass before you train on them
📰

Shortcut judges: a cheap test your preference pairs should pass before you train on them

DEV Community: machinelearning·Femi Ogunsanya·2 days ago
#oFyvMki5
Reading 0:00
15s threshold

I'm building evaluation adapters (small reward models) for Telegraph, a network that routes AI requests to miners and scores their answers. Disclosure: I'm competing in Telegraph Hackathon Season II, which has prizes.

A preference pair is a prompt, a better answer and a worse one. A reward model trained on pairs learns whatever separates the two answers. If something cheaper than the skill you want separates them, it learns that instead, and your accuracy number measures the shortcut.

The test

Before training, run rules that pick an answer without checking anything:

pick the longer answer

pick the answer with more words in common with the prompt (TF-IDF cosine)

pick the answer with more of an obvious feature (alarm words, a verdict word)

If any of them is far above 50%, fix the pairs first.

Two cases from this week

Another builder (AdversarialEval, in the Telegraph Discord) found a keyword-count judge got every one of their severity pairs right, because all the worse answers were downgrades.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.