Menu

1 / 2

How we measured AI writing across arXiv, and where the measurement breaks

Hacker News·4 months ago
#uhBmSv6i
#unslop#papers#control#field#detector
Reading 0:00
15s threshold

We scored the full text of 12,750 arXiv papers and found that about a third of new ones read as machine-written. Here is the method, the results, and an honest account of the limitations.

Share of new arXiv papers flagged as machine-written, 2021 to 2026, at a threshold calibrated so pre-ChatGPT papers flag at 0.4%. The eight slate points are the pre-LLM control months; the band is a bootstrap 95% interval.

Share of papers flagged as machine-written by field, over the 12 months to July 2026 (about 300 papers per field).

A false-positive floor

There is a genre of headline that says "N% of X is now AI," and most are not worth reading, because the detector behind the number also flags some share of genuine human writing. If a tool marks 40% of new papers as machine-written but also marks 20% of papers written before ChatGPT existed, the real story is the 20% nobody mentioned.

So we built the study around that objection.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.