Menu

Prompt Diff, Eval Drift: Lock-File the Prompt Half of Your AI C++ Eval Harness
📰

Prompt Diff, Eval Drift: Lock-File the Prompt Half of Your AI C++ Eval Harness

DEV Community: cpp·Finley Li·10 days ago
#zAEMF2Zq
#devtocpp#devto#dev#prompt#lock
Reading 0:00
15s threshold

A typical failure looks like this. Someone tightens one sentence in the system prompt — prefer becomes must — and the next sweep of the C++ golden set drops fourteen points. The model gets blamed, a migration gets proposed, and nobody notices that the harness never recorded which prompt produced the baseline.

The numbers in that sketch are illustrative, not a benchmark. The failure shape, though, is common: eval harnesses version their golden cases and their graders, and leave the prompt as an untracked string in a YAML file.

This post covers the missing half. It is a lock-file discipline for everything on the prompt side that can move a verdict, plus a paired replay that tells you whether a drop came from the prompt, the model, or the box.

Why the prompt is the unstable half

A C++ patch harness has two halves. The execution half — compiler, standard flag, sanitizers, test runner — is usually pinned, because someone got burned by a -O2 -only failure once.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.