How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the answer: It uses a Mixture-of-Experts ( MoE ) architecture that selects only a subset of its parameters for each token .

There are two dominant model architectures: Dense model and MoE. How a model organizes its parameters matters as much as how many it has. The right choice affects throughput, memory cost, and serving complexity more than raw parameter count does. Therefore, choosing between them comes down to your deployment constraints.

Think of the difference like two engines with the same total displacement. One fires every cylinder on every cycle; the other activates only the cylinders it needs.

This post explains:

How dense and MoE architectures work

How they affect performance

When to choose each one

Dense models vs. MoE models

In simple terms, dense and MoE models differ in how they use their parameters.…