Your weekly refresh finishes successfully. Then the chatbot loses answers from pages nobody edited.
The suspicious line is downstream: replace_index(dataset) . Once you enable crawlOnlyPagesChangedSince , that export contains changed pages. Unchanged pages are absent by design.
Lab note 047: can we make corpus refreshes cheaper without interpreting silence as permission to delete? The catch comes first: a changed-page delta cannot establish which documents were deleted.
What can an incremental export prove?
Our site crawler uses HTTP validators, sitemap lastmod , and content hashes to identify changes. That helps decide what to return. It doesn't turn an omitted URL into a tombstone.
There's another boundary: maxCrawlPages defaults to 20 and limits stored pages and files.…