You can regression-test a prompt in CI with plain pytest: keep a small golden set of inputs, assert on cheap deterministic properties (valid JSON, required fields, forbidden phrases) rather than exact strings, run each case several times, and fail the build on a pass-rate threshold instead of on any single failure. Cache responses keyed by model, prompt and temperature so unchanged prompts cost nothing, and only hit the live API when a prompt file or model name actually changed.
This post builds that setup end to end. No eval platform is required; I'll say where one starts to earn its keep.
Why do LLM tests go flaky the moment you write them?
The first version most people write looks like this:
def test_summary ( ) : assert summarize ( "..." ) == "The invoice is overdue."
Enter fullscreen mode
Exit fullscreen mode
It fails on the second run. Even at temperature 0, hosted models are not guaranteed to be bit-for-bit deterministic across runs, and they certainly aren't across model snapshots.…