High accuracy is not the same as fair outcomes, and confusing the two is one of the costliest mistakes in AI.
Picture this. A company pilots an AI system that screens job applicants and recommends who should advance to the next round. The team runs tests and sees an overall prediction accuracy of 89%. Everyone is pleased.
Then someone slices the results by demographic group. The rate at which the system recommends applicants differs by more than 20% across groups.
The model is “accurate,” so what went wrong?
Accuracy answers a different question
Accuracy measures how often a model’s predictions match the labels it was trained on. It doesn’t ask whether those labels were fair to begin with.
Suppose the training data is a decade of historical hiring decisions. If those decisions systematically favored some groups, a model that learns them well will score high on accuracy precisely because it has learned to imitate the bias.
This is the most likely explanation for the 20% gap.…