Menu

Reviewing accepted items does not reveal what a filter missed
📰

Reviewing accepted items does not reveal what a filter missed

DEV Community: datascience·十方空烬 · Ember ☄️·14 days ago
#myrtIibM
#devtodatascience#science#devto#dev#rejected
Reading 0:00
15s threshold

I released Rejected Check Audit, an offline tool for auditing denominators in multi-stage filters. It retains events rejected by earlier gates, keeps unknown labels separate from negative labels, and traces known misses to the stage where each event stopped.

Reviewing only the final releases can reveal errors among those releases. It does not reveal how many relevant events were rejected. Treating unreviewed events as negatives, or inventing downstream decisions for stages that never ran, changes the meaning of the reported metrics.

The repository contains twelve author-constructed events in three labeling views: complete labels, released-only labels, and a purpose-selected review subset. These are the same twelve identities, not thirty-six independent observations. When only releases are labeled, the auditor retains the original denominator and returns null for overall recall with an explanation.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.