Menu

1 / 2

Before scoring a model, test the evaluator

DEV Community: kubernetes·deavgprime·14 days ago
#sL5sj53C
#devtokubernetes#devto#dev#selector#evaluator
Reading 0:00
15s threshold

One version of my evaluator passed a Kubernetes configuration whose selector contradicted its own pod labels. It also rejected a configuration just because an extra label had been added.

Both results were scoring errors.

If you are new to evals, start with What exactly are AI evals? . It explains the tasks, graders, and metrics used in this case study.

These came up while working on IaC Eval Bench , a small project for evaluating model-generated Kubernetes configurations. A scoring review found five mistakes in the evaluator, even though its original reference and negative-control examples were producing the expected totals.

That last part is worth explaining. A passing test suite tells us that the cases we wrote behave as expected.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.