Note

This is a shortened version of the original writing on kig.re — my personal blog.

Here is what a real lead typed into a tax-preparation intake form:

“Got married in June, haven’t figured out how we’re filing yet. There’s an IRS letter somewhere in a kitchen drawer I haven’t opened. I also deliver groceries on weekends.”

Only one fact in that paragraph is clear: the weekend side gig. A model read it anyway and set filing_status to married_jointly .

Helpful. Wrong. The form then skipped the filing-status question and treated a guess as settled. That is the failure I care about most, and it is invisible if you only eyeball a few outputs.

The vague tax-preparation case: the lead's input, the expected JSON with filing_status null, gpt-4o-mini hallucinating married_jointly at 78.8%, and deepseek-v4.1-flash-1017 scoring 100% with empty diagnostics.

I built llm-evals so I would stop writing scripts/eval.rb for the fourth time. It is a Ruby CLI for multi-dataset, multi-case evaluation suites.…