Menu

How to Regression-Test LLM Prompts in CI Without Flaky Tests or a Surprise Bill
📰

How to Regression-Test LLM Prompts in CI Without Flaky Tests or a Surprise Bill

DEV Community: cicd·Libme·2 days ago
#7ZRl1Ae9
#devtocicd#devto#dev#prompt#model
Reading 0:00
15s threshold

You can regression-test a prompt in CI with plain pytest: keep a small golden set of inputs, assert on cheap deterministic properties (valid JSON, required fields, forbidden phrases) rather than exact strings, run each case several times, and fail the build on a pass-rate threshold instead of on any single failure. Cache responses keyed by model, prompt and temperature so unchanged prompts cost nothing, and only hit the live API when a prompt file or model name actually changed.

This post builds that setup end to end. No eval platform is required; I'll say where one starts to earn its keep.

Why do LLM tests go flaky the moment you write them?

The first version most people write looks like this:

def test_summary ( ) : assert summarize ( "..." ) == "The invoice is overdue."

Enter fullscreen mode

Exit fullscreen mode

It fails on the second run. Even at temperature 0, hosted models are not guaranteed to be bit-for-bit deterministic across runs, and they certainly aren't across model snapshots.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.