You change one line of a prompt. The demo still works. You merge. Two days later a user screenshots a nonsense answer and asks what happened.

What happened: nothing caught the regression, because nothing ran the prompt against real cases before merge.

Unit tests do not cover this. A prompt is not a function with one right answer - it has a distribution of acceptable answers, and the only cheap way to pin that distribution down is a small golden dataset plus a runner that scores every change against it.

The 20-minute setup

Write 10-30 cases from real traffic. Each case: the input, plus what a good answer must contain (or must not contain). JSONL is fine:

{ "input" : "refund request, order late by 3 weeks" , "must_include" : [ "refund" , "apolog" ] , "must_not_include" : [ "as an AI" ] } { "input" : "ask for discount on enterprise plan" , "must_include" : [ "annual" ] , "max_words" : 120 }

Enter fullscreen mode

Exit fullscreen mode

Run your app against the cases on every PR.…