A typical failure looks like this. Someone tightens one sentence in the system prompt — prefer becomes must — and the next sweep of the C++ golden set drops fourteen points. The model gets blamed, a migration gets proposed, and nobody notices that the harness never recorded which prompt produced the baseline.
The numbers in that sketch are illustrative, not a benchmark. The failure shape, though, is common: eval harnesses version their golden cases and their graders, and leave the prompt as an untracked string in a YAML file.
This post covers the missing half. It is a lock-file discipline for everything on the prompt side that can move a verdict, plus a paired replay that tells you whether a drop came from the prompt, the model, or the box.
Why the prompt is the unstable half
A C++ patch harness has two halves. The execution half — compiler, standard flag, sanitizers, test runner — is usually pinned, because someone got burned by a -O2 -only failure once.…