A broken queue job throws. A bad migration stops the deploy. A RAG feature that gets worse does neither: it keeps answering confidently and keeps citing sources. They are just the wrong sources, and you find out when a user does.
I built laravel-rag , an open-source Composer package, so that kind of regression fails a build instead. Its rag:eval command runs golden questions against your corpus, scores recall and MRR, and with --ci exits non-zero when a score drops below a floor you set. Version 0.3.0 rebuilt the package on the official Laravel AI SDK, and this tutorial is written for it. The laravel-rag case study covers why it exists; this post goes from install to a red GitHub Actions check.
Why retrieval regressions never throw an error
Three ordinary changes cause most of them: a different chunk size, a cheaper embedding model, or a few hundred new documents crowding the old ones out of the top results.β¦