TL;DR — We built ja-ocr-error-detect-v1, a detector for OCR errors in prewar Japanese printed texts, and we are not releasing it. Inside its synthetic training distribution it scored character F1 0.64. On real OCR output — 80 pages of National Diet Library modern materials, used for evaluation only — it scored F0.5 0.078 and caught about 2% of the actual errors. Three generations of fixes later it still missed the release bar we had set in advance (F0.5 ≥ 0.20 with at most 1.0 false flags per 1,000 characters), so the project is frozen. This is the full record of what we measured.
The synthetic-to-real gap was not a gap; it was a cliff. Character F1 0.64 on synthetic data became F0.5 0.078 on real OCR text, detecting roughly 2% of real errors. The errors we had synthesized were not the errors prewar print actually produces.
The untrained baseline beat the trained model. Before any fine-tuning, the base model's masked-language-model signal scored PR-AUC 0.137 at spotting real errors.…