Monday AI Insight. Most of us add few-shot examples to a prompt the same way: grab three or four that look right, paste them in, ship. It works often enough that we never ask what the model is actually learning from them.
The research answer is a little uncomfortable: the model mostly learns the shape of the task from your examples, not the right answers. And it picks up your mistakes in label balance and ordering along the way.
Here's what three well-known papers found, and the practical rules I take from them.
1. Examples teach the format and the label space
Min et al. (2022), Rethinking the Role of Demonstrations , tested 12 models (including GPT-3) on classification and multiple-choice tasks. They swapped the correct labels in the demonstrations for random labels , and performance barely moved: drops were mostly in the 0β5% absolute range.
What did matter:
The label space : showing the model which answers are allowed.
The input distribution : examples that look like the real inputs.β¦