I thought my RAG pipeline was solid.

We had a notebook that retrieved the right chunks. Answers looked grounded. Ingest scores looked healthy. The demo never failed in front of stakeholders.

Then a small change went out: we tightened chunk size to "improve precision."

CI stayed green. Unit tests stayed green.

Two weeks later support tickets started — answers that skipped the corpus, citations that pointed at the wrong doc, and one nasty case where a low-quality upload slipped ingest because trust_score still looked fine.

That was the moment I stopped trusting notebook confidence.

What actually broke

The failures were not dramatic. They were quiet.

Recall dropped after the chunk tweak, but nothing in CI measured retrieval quality. The model still answered fluently, so grounding gaps hid behind confident prose. Ingest trusted a score that was not the same as "safe to serve."

We were testing code.…