TL;DR: Give every marketplace page snapshot a stable version, derive each chunk ID from that version plus its position, and make the bulk write an upsert. This is the least complex design that stops retries from increasing vector count. Log runId , sourceId , sourceVersion , chunkId , and the write outcome together. Then a repeated search hit becomes a traceable ingestion event instead of a mysterious retrieval problem.
Start with the write path. Re-ranking or filtering can hide duplicate chunks, but neither reduces index growth. For a page-change watcher, that distinction matters: a retry should cost another request, not another permanent vector.
Retries happen.
Strategy Pick this when Retry behavior Index-cost effect Main limit
Random vector IDs Almost never for replayable ingestion Inserts another row Cost grows with every replay No stable identity to inspect
Deterministic chunk IDs The same snapshot may be retried Overwrites the same logical row Stable for identical input Requires stable…