Menu

The Billion-Dollar Model That Can't Count: Why Scaling Laws Hit a Wall
📰

The Billion-Dollar Model That Can't Count: Why Scaling Laws Hit a Wall

DEV Community: ai·VelocityAI·5 days ago
#7QWrwpuD
#devtoai#devto#dev#model#wall
Reading 0:00
15s threshold

GPT-4 can pass the bar exam, score in the 90th percentile on the SAT, and write a coherent essay on the themes of Moby-Dick. It also, until recently, struggled to reliably count the number of r’s in “strawberry.” A model trained on trillions of tokens, at a cost estimated in the hundreds of millions of dollars, failed at a task that a seven-year-old can do. The r’s in strawberry became a meme, and the meme obscured something important: it was not an isolated glitch. It was a symptom of a structural limit that the scaling paradigm has been papering over for years.

Here’s the claim I want to test: the scaling laws are real, but they are laws about a specific kind of capability, and they do not extrapolate to the capabilities that matter most. Bigger models get better at the things that benchmarks measure.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.