Menu

Marketplace Page Alerts: Debug Duplicate Chunks Across Vector Results
📰

Marketplace Page Alerts: Debug Duplicate Chunks Across Vector Results

DEV Community: typescript·VaughnKnight3189·1 day ago
#BYsKNDGZ
Reading 0:00
15s threshold

TL;DR: Give every marketplace page snapshot a stable version, derive each chunk ID from that version plus its position, and make the bulk write an upsert. This is the least complex design that stops retries from increasing vector count. Log runId , sourceId , sourceVersion , chunkId , and the write outcome together. Then a repeated search hit becomes a traceable ingestion event instead of a mysterious retrieval problem.

Start with the write path. Re-ranking or filtering can hide duplicate chunks, but neither reduces index growth. For a page-change watcher, that distinction matters: a retry should cost another request, not another permanent vector.

Retries happen.

Strategy Pick this when Retry behavior Index-cost effect Main limit

Random vector IDs Almost never for replayable ingestion Inserts another row Cost grows with every replay No stable identity to inspect

Deterministic chunk IDs The same snapshot may be retried Overwrites the same logical row Stable for identical input Requires stable…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.