Security feeds publish lists of malicious packages, and dependency scanners turn those lists into
red alerts in your CI. How often is a flag right? Nobody publishes that number per feed, so we
started measuring it, with the method written down and committed before we looked at a single
result. We also ran four open-source scanners on known malware and on a week of new uploads, to see
what a flag costs at a registry's real volume. For completeness of the analysis, we also measured
what each feed's records show and how each feed answers the people who dispute them.
Our interest, first. One of the feeds below listed nine of our own PyPI packages as malware; it has since reviewed all
nine, withdrawing one and reclassifying eight as vulnerabilities. That is why we looked. They are counted like every other row, each feed gets its numbers a week before
this post, and their answers are part of it.…