Rules-Only or LLM Judge? A Scoring Harness for the Second Reviewer of Agent SQL
Two engineers are looking at the same pull request. One of them added llm_judge.py , a second model call that re-reads every SQL draft before the read-only warehouse role ever sees it. The other objects on latency, cost, and nondeterminism, and answers that the rule engine missed two incidents last quarter. Neither engineer has a number, so the thread keeps recycling the same two anecdotes in slightly different wording.
This article replaces that argument with a harness. You label a set of drafts once, run both reviewers over it, and compare confusion matrices instead of instincts. Everything below is runnable, and the only assumption is that you already have agent-written SQL reaching a database whose tables you can describe.
1. Where this debate sits relative to the earlier ones
Earlier posts on this account compared parser gates with runtime guards, and shadow replay with a read-only canary.…