We had a reconciliation job that matched our orders to the acquirer's settlement report by calendar day. It worked for weeks. Then finance started seeing a handful of unmatched transactions every morning, always clustered in the same two-hour window in the evening.
Nothing was missing. The transactions showed up the next day, in the next report.
The cause: we bucketed by our own created_at in UTC midnight to midnight. The acquirer's settlement batch closes at a fixed cutoff that is not midnight UTC. In our case it was a few hours earlier. A payment captured at 21:40 UTC belonged to tomorrow's batch for the acquirer and to today's for us. So every day, a few transactions were "missing" from one side and "extra" on the other, and then self-healed 24 hours later. That is why nobody could reproduce it on demand.
The fix was boring. We stopped deriving the batch window ourselves.…