Menu

Freezing a model that caught 2% of real errors: OCR error detection for prewar Japanese print
📰

Freezing a model that caught 2% of real errors: OCR error detection for prewar Japanese print

DEV Community: machinelearning·Japan Open Data Lab·about 6 hours ago
#x6ILgLaU
Long article (1076 words) — no AI summary yet.
Reading 0:00
15s threshold

TL;DR — We built ja-ocr-error-detect-v1, a detector for OCR errors in prewar Japanese printed texts, and we are not releasing it. Inside its synthetic training distribution it scored character F1 0.64. On real OCR output — 80 pages of National Diet Library modern materials, used for evaluation only — it scored F0.5 0.078 and caught about 2% of the actual errors. Three generations of fixes later it still missed the release bar we had set in advance (F0.5 ≥ 0.20 with at most 1.0 false flags per 1,000 characters), so the project is frozen. This is the full record of what we measured.

The synthetic-to-real gap was not a gap; it was a cliff. Character F1 0.64 on synthetic data became F0.5 0.078 on real OCR text, detecting roughly 2% of real errors. The errors we had synthesized were not the errors prewar print actually produces.

The untrained baseline beat the trained model. Before any fine-tuning, the base model's masked-language-model signal scored PR-AUC 0.137 at spotting real errors.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.