I wanted a clean table of Egypt's pharaohs: who ruled, in which dynasty, for how long, and which pyramid belongs to whom. Wikidata has most of it, but "most of it, as linked data" is a long way from a table you can trust. So I built it the way I would build any production dataset: extract, clean, validate, and keep a record of every value the pipeline refused.

The result is three tables (513 pharaohs, 38 dynasties, 84 pyramids), published on Kaggle and Hugging Face , with the code on GitHub .

The pipeline

extract.py three SPARQL queries against query.wikidata.org, one a minute -> raw JSON + a manifest with the query text and the retrieval time build.py raw rows -> tables (dates, precision flags, one row per entity, chronology join), out-of-range dates nulled and logged validate.py pandera schemas for every table; the build fails if a check fails

Enter fullscreen mode

Exit fullscreen mode

Two habits from production work made the biggest difference.

Keep the raw pull and its provenance.…