A dense transformer pays for every parameter on every token. Double the model, double the compute, no matter how easy the token is. Mixture of Experts (MoE) breaks that bargain. It keeps a huge pool of parameters available, but activates only a small subset for each token. The result is a model that can be far larger in total size while costing roughly the same per token as a much smaller dense model. That decoupling of capacity from compute is the core idea, and it explains both the excitement around MoE and most of its failure modes.

This article explains how routing works, why the load balancing loss matters, what happens when routing collapses, and when MoE is the wrong choice.

The intuition: specialists, not generalists

Think of a hospital. You would not ask every doctor to examine every patient. You route a cardiac case to cardiology and a fracture to orthopaedics. The hospital is large, but each patient sees only a few specialists.…