I expected small local models to be worse than a hosted decision API at Japanese classification. I did not expect most of the gap to be something other than judgement. On the hardest of three tasks one model scored 10.8% overall while getting 27 of the 29 answers it produced right. The other 221 rows it never answered.

What I ran, and what the accuracy number counted

Three Japanese tasks, 250 identical rows each. livedoor is 9-class news-topic classification (mean 1,308 characters), rakuten is 2-class review polarity (165), chabsa is 3-class financial-sentence polarity (100). Chance is 11.1%, 50.0% and 33.3%. Every system got the same prompt asking for one label as JSON, and a row counted correct only if the parsed label matched gold.…