The Judge Reads Tokens: An Adversarial Evaluation of an Eval Toolkit's Own Scorers
Abstract. Every project in this portfolio got an adversarial eval suite; this one is the eval toolkit itself, so the system under attack is the scoring pipeline β the mock LLM judge and the deterministic scorers ( contains , regex ) that grade other projects' outputs. I aimed three attack families at it: outputs engineered to game the judge (negation, keyword echo, invented facts appended to true ones, fake citations), text addressed to the judge inside the candidate output (instructions, a pre-written verdict, and a smuggled replacement for the machine-readable CASE block the mock judge actually reads), and substring/regex exploits against the deterministic scorers β plus bias probes for length, word order and wording, and a human-labelled agreement slice to compare against the benign agreement set.β¦