Menu

Null rate data quality check in SQL, per column
📰

Null rate data quality check in SQL, per column

DEV Community: postgres·Refael Dakar·9 days ago
#AFGaeYgn
#devtopostgresql#devto#dev#count#null
Reading 0:00
15s threshold

A dashboard goes wrong quietly in two ways. The rows stop arriving, which a freshness or row count check catches. Or the rows keep arriving and one column fills with nulls. Nothing errors, the load job is green, and revenue by coupon quietly drops to zero.

A null rate check is the cheap answer to the second one. Plain Postgres SQL below; the full write-up (every column at once, per-role tolerances, backtesting) is on my blog.

Null rate for one column

SELECT count ( * ) AS total , round ( 100 . 0 * count ( * ) FILTER ( WHERE email IS NULL ) / NULLIF ( count ( * ), 0 ), 2 ) AS email_null_pct FROM orders WHERE created_at >= now () - interval '28 days' ;

Enter fullscreen mode

Exit fullscreen mode

NULLIF(count(*), 0) turns an empty window into a null instead of a division error, so the check doesn't crash on exactly the day the table is empty.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.