TL;DR
An ID tells you where a record came from, not what's in it. If the same file can arrive from two places, compare the file itself.
Two owners can publish the exact same file, and both copies are real. A duplicate check has to include who the file belongs to.
When the duplicate check itself breaks, decide on purpose what happens next. We chose for our main save jobs to save everything, because losing a real document costs more than keeping an extra copy.
If two checks answer "have we seen this before?", they must agree on what counts as seen, including whether records that were deleted, or already flagged as copies of another record, still count.
Passing tests don't prove a new step runs. Walk one real day of input through every filter in front of it.
Our pipeline checked every new document against the ones it already had. It still saved the same PDF twice. The check was working fine, it was just checking the wrong thing.
You've probably seen this at home.…