One version of my evaluator passed a Kubernetes configuration whose selector contradicted its own pod labels. It also rejected a configuration just because an extra label had been added.
Both results were scoring errors.
If you are new to evals, start with What exactly are AI evals? . It explains the tasks, graders, and metrics used in this case study.
These came up while working on IaC Eval Bench , a small project for evaluating model-generated Kubernetes configurations. A scoring review found five mistakes in the evaluator, even though its original reference and negative-control examples were producing the expected totals.
That last part is worth explaining. A passing test suite tells us that the cases we wrote behave as expected.…