Mixture-of-Depths (MoD): Dynamically routing computational depth per token—allowing complex tokens to dive into deep attention layers while simpler tokens bypass via residual skip connections.
For years, standard transformer architectures have processed input tokens uniformly. Whether a model processes a simple stop word like "the" or a dense mathematical term like "eigenvector", every single token passes through every layer of the network, incurring an identical computational cost.
This static depth model creates a massive efficiency bottleneck. In natural language, not all tokens require the same depth of processing to understand their context and predict the next token.
To break past this static constraint, AI researchers have introduced Mixture-of-Depths (MoD)—an architectural breakthrough that dynamically allocates compute on a per-token basis.
What is Mixture-of-Depths?
Mixture-of-Depths (MoD) is a neural architecture variant that allows transformers to dynamically decide which tokens should…