Menu

Your RAG Has No Idea What Year It Is: A Frozen Benchmark for Version Leakage
πŸ“°

Your RAG Has No Idea What Year It Is: A Frozen Benchmark for Version Leakage

DEV Community: nlpΒ·RaihanΒ·1 day ago
#o60XEdvV
#devtonlp#devto#dev#asked#questions
Reading 0:00
15s threshold

I froze nine yearly revisions of federal acquisition regulations into one corpus, asked 700 questions "as of" specific dates, and watched plain hybrid RAG answer from the wrong year 80% of the time. Date handling fixes it β€” with one honest failure mode.

Scope note, first paragraph as promised: FAR/DFARS Title 48 only, year-granularity revisions 2017–2025, US public-domain text, bge-m3 + BM25 retrieval, rule-based scoring, no LLM judge, no generator trained yet. NOT legal advice β€” a measurement study, not counsel.

Timeless corpus, wrong answers

Regulations change. Between 2017 and 2025, about 9,600 section-year pairs in Title 48 changed text β€” with a real spike around 2019–2020 (2,318 pairs). A RAG system that ignores dates retrieves across all nine years and answers with whatever ranks highest. Usually that's the newest text.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.