Menu

Same Row Count, Missing Records: 4 Ways a Count Check Can Fool You
📰

Same Row Count, Missing Records: 4 Ways a Count Check Can Fool You

DEV Community: database·KAREN·about 9 hours ago
#q9AwKTYK
#devtodatabase#devto#dev#count#destination
Reading 0:00
15s threshold

A common check after a batch job is to compare two numbers: how many records went in, and how many came out. If they match, the job is marked healthy.

That check is useful as an alarm. But it is not proof. Two lists can have the same length and still hold different records.

A tiny made-up example

The run was supposed to deliver four orders:

Expected: A , B , C , D

Found in the destination: A , A , C , E

Count check: 4 expected, 4 found. Looks fine.

Compare by ID instead:

missing : B , D

duplicate : A (written twice)

unexpected : E (not part of this run)

Three problems, and the count hid all of them.

4 ways the count can match while records are missing

1. A retry wrote duplicates that cancel out a drop

One batch failed halfway and was retried. The retry wrote some rows a second time, while a different set of rows never made it. The extra copies fill the gap in the total.

2.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.