Menu

The Judge Reads Tokens: An Adversarial Evaluation of an Eval Toolkit's Own Scorers
πŸ“°

The Judge Reads Tokens: An Adversarial Evaluation of an Eval Toolkit's Own Scorers

DEV Community: machinelearningΒ·Tushar KumarΒ·9 days ago
#Gm8r5d6l
#devtomachinelearning#education#devto#dev#judge
Reading 0:00
15s threshold

The Judge Reads Tokens: An Adversarial Evaluation of an Eval Toolkit's Own Scorers

Abstract. Every project in this portfolio got an adversarial eval suite; this one is the eval toolkit itself, so the system under attack is the scoring pipeline β€” the mock LLM judge and the deterministic scorers ( contains , regex ) that grade other projects' outputs. I aimed three attack families at it: outputs engineered to game the judge (negation, keyword echo, invented facts appended to true ones, fake citations), text addressed to the judge inside the candidate output (instructions, a pre-written verdict, and a smuggled replacement for the machine-readable CASE block the mock judge actually reads), and substring/regex exploits against the deterministic scorers β€” plus bias probes for length, word order and wording, and a human-labelled agreement slice to compare against the benign agreement set.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.