Menu

Post image 1
Post image 2
1 / 2
0

How we measured AI writing across arXiv, and where the measurement breaks

Hacker News·2 months ago
#uhBmSv6i
#unslop#papers#control#field#detector#flag
Reading 0:00
15s threshold

We scored the full text of 12,750 arXiv papers and found that about a third of new ones read as machine-written. Here is the method, the results, and an honest account of the limitations. Share of new arXiv papers flagged as machine-written, 2021 to 2026, at a threshold calibrated so pre-ChatGPT papers flag at 0.4%. The eight slate points are the pre-LLM control months; the band is a bootstrap 95% interval. Share of papers flagged as machine-written by field, over the 12 months to July 2026 (about 300 papers per field). A false-positive floor There is a genre of headline that says "N% of X is now AI," and most are not worth reading, because the detector behind the number also flags some share of genuine human writing. If a tool marks 40% of new papers as machine-written but also marks 20% of papers written before ChatGPT existed, the real story is the 20% nobody mentioned. So we built the study around that objection.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More