My label distribution chart was perfect. 38% billing, 22% login, 15% shipping, and so on down the line, exactly what I expected from six months of support tickets. The classifier I trained on those labels hit a validation accuracy I was happy with. I almost shipped it.

Then I hand-labeled 200 tickets for a gold set, compared them against Claude's labels, and got 41.5% agreement. Claude wasn't that bad at reading support tickets. My pipeline was. The Claude Batch API results came back out of order, I matched them to my inputs by position, and 2,940 of my 5,000 labels ended up attached to the wrong ticket.

Every dashboard I had looked fine, and here's why: every dashboard I had was a histogram.

TL;DR

The Claude Message Batches API does not return results in the order you submitted requests. The docs say so. I didn't read that line.

Zipping results with inputs by index silently mislabels data.…