Menu

Correct, Cited, and Two Years Out of Date: Measuring Version Leakage in RAG
πŸ“°

Correct, Cited, and Two Years Out of Date: Measuring Version Leakage in RAG

DEV Community: nlpΒ·RaihanΒ·2 days ago
#9yZ8cwC9
#devtonlp#devto#dev#version#wrong
Reading 0:00
15s threshold

I built RAG over nine yearly revisions of federal acquisition regulations and asked it questions "as of" specific dates. Plain RAG answers from the wrong version 80% of the time β€” fluently, with citations. Date handling fixes it without hurting recall.

Scope note, first paragraph as promised: FAR/DFARS only, year-granularity revisions (2017–2025), US public-domain text, one 1.8B local generator, greedy decoding, no LLM judge (containment is string matching). NOT legal advice β€” a measurement study, not counsel.

The problem nobody benchmarks

Regulations change. The micro-purchase threshold moves, clauses get renumbered, paragraphs appear and vanish. A RAG system that treats its corpus as timeless will answer a 2022 question with 2024 text β€” correctly cited, completely wrong. I call it version leakage , and before this project I couldn't find an open benchmark that measures it over a corpus with official point-in-time history.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.