Menu

Rules-Only or LLM Judge? A Scoring Harness for the Second Reviewer of Agent SQL
📰

Rules-Only or LLM Judge? A Scoring Harness for the Second Reviewer of Agent SQL

DEV Community: database·Morgan Li·8 days ago
#EBP3X1Th
#devtodatabase#devto#dev#judge#rules
Reading 0:00
15s threshold

Rules-Only or LLM Judge? A Scoring Harness for the Second Reviewer of Agent SQL

Two engineers are looking at the same pull request. One of them added llm_judge.py , a second model call that re-reads every SQL draft before the read-only warehouse role ever sees it. The other objects on latency, cost, and nondeterminism, and answers that the rule engine missed two incidents last quarter. Neither engineer has a number, so the thread keeps recycling the same two anecdotes in slightly different wording.

This article replaces that argument with a harness. You label a set of drafts once, run both reviewers over it, and compare confusion matrices instead of instincts. Everything below is runnable, and the only assumption is that you already have agent-written SQL reaching a database whose tables you can describe.

1. Where this debate sits relative to the earlier ones

Earlier posts on this account compared parser gates with runtime guards, and shadow replay with a read-only canary.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.