Menu

A Changed-Page Export Can Quietly Shrink Your RAG Index
📰

A Changed-Page Export Can Quietly Shrink Your RAG Index

DEV Community: nlp·vexaloom-labs·13 days ago
#Uv2k5vFe
#devtonlp#devto#dev#doc_id#document
Reading 0:00
15s threshold

Your weekly refresh finishes successfully. Then the chatbot loses answers from pages nobody edited.

The suspicious line is downstream: replace_index(dataset) . Once you enable crawlOnlyPagesChangedSince , that export contains changed pages. Unchanged pages are absent by design.

Lab note 047: can we make corpus refreshes cheaper without interpreting silence as permission to delete? The catch comes first: a changed-page delta cannot establish which documents were deleted.

What can an incremental export prove?

Our site crawler uses HTTP validators, sitemap lastmod , and content hashes to identify changes. That helps decide what to return. It doesn't turn an omitted URL into a tombstone.

There's another boundary: maxCrawlPages defaults to 20 and limits stored pages and files.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.