The Canadian Food Inspection Agency (CFIA) tests food for infants and children every year in its Children's Food Project, and it publishes one per-sample CSV per year on open.canada.ca, next to a report page with printed tables. That is rare and valuable: anyone can check the printed numbers against the raw data. So we did, for aflatoxin M1 (a mould toxin that reaches milk) in six years of files, and for lead in six more.
Every check was pre-registered: the predictions and the grading code were public on GitHub before any result value was read. A sealed compute job then graded them on SHA-256-pinned copies of the files, and a second sealed job later re-ran an independent test against the published results.
Most of it reproduces
The good news first. For aflatoxin M1 (2012-13 to 2022, 468 samples), the files reproduce every figure CFIA printed. For lead (2017 to 2022, 979 samples), they reproduce every row of every printed table: samples tested, negative, positive and the maximum.…