What Are Emergent Abilities?
Emergent abilities are skills or behaviors that a system unexpectedly develops once it becomes complex enough—even though these abilities weren’t programmed in directly.
Take water as an example. A single hydrogen or oxygen atom doesn’t have the property of “wetness.” But combine billions of them to make water, and wetness emerges. Wetness can’t be traced to any single atom; it shows up only when there are enough atoms interacting together.
In large language models (LLMs), emergent abilities show up in a similar way. Nobody codes in a “math skill” setting or flips a translation switch. Instead, once a language model crosses a certain size or training level, it suddenly starts demonstrating new skills all by itself. For many engineers, this is surprising—even counterintuitive.…