Menu

Few-shot examples teach format, not facts: how to pick, balance and order them
πŸ“°

Few-shot examples teach format, not facts: how to pick, balance and order them

DEV Community: machinelearningΒ·M.T.RamkrushnaΒ·14 days ago
#wgZH85S9
#devtomachinelearning#education#devto#dev#examples
Reading 0:00
15s threshold

Monday AI Insight. Most of us add few-shot examples to a prompt the same way: grab three or four that look right, paste them in, ship. It works often enough that we never ask what the model is actually learning from them.

The research answer is a little uncomfortable: the model mostly learns the shape of the task from your examples, not the right answers. And it picks up your mistakes in label balance and ordering along the way.

Here's what three well-known papers found, and the practical rules I take from them.

1. Examples teach the format and the label space

Min et al. (2022), Rethinking the Role of Demonstrations , tested 12 models (including GPT-3) on classification and multiple-choice tasks. They swapped the correct labels in the demonstrations for random labels , and performance barely moved: drops were mostly in the 0–5% absolute range.

What did matter:

The label space : showing the model which answers are allowed.

The input distribution : examples that look like the real inputs.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.