Menu

Turning 12 benchmark write-ups into a citable dataset (with a DOI) in one afternoon
📰

Turning 12 benchmark write-ups into a citable dataset (with a DOI) in one afternoon

DEV Community: datascience·Xplormity·3 days ago
#qB0iKRDI
#devtodatascience#science#devto#dev#write
Long article (640 words) — no AI summary yet.
Reading 0:00
15s threshold

We publish each stage of NewsShorts-Bench as a long write-up: the question, the systems we ran, a results table, the method and what broke. Twelve stages in, the tables were only readable as web pages. Nobody could load them into pandas, and nobody could cite them properly.

So we archived the results as a dataset with a DOI. It took one afternoon. Here is the whole process, because most of it applies to any project that publishes numbers in Markdown.

1. Export from the pages you already publish

Our write-ups are MDX files with the results in frontmatter (name, value, confidence interval, extra columns). The rule we kept: the dataset must be generated from the same files the site renders , never retyped. If a number changes, both change.

import csv , glob , yaml

for path in sorted ( glob . glob ( "content/benchmarks/*.mdx" ) ) : fm = yaml . safe_load ( open ( path , encoding = "utf-8" ) . read ( ) . split ( "---" ) [ 1 ] ) keys = [ c [ "key" ] for c in fm .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.