Mixture-of-Experts (MoE) is the architecture behind a striking trick: a model with a trillion parameters that runs at roughly the cost of a much smaller dense one. The promise is easy to state and hard to engineer — more capacity, similar compute.
From dense to sparse
In a standard transformer, every token flows through the same feed-forward block; every parameter fires for every token. That's wasteful — most tokens don't need most of the network. An MoE layer replaces that single block with many experts plus a small router.
For each token, the router scores the experts and activates only the top k (often two). Capacity lives in the full set of experts; cost is paid only for the few that run.
Where the difficulty hides
- Load balancing. Left alone, the router collapses onto a few favorite experts. An auxiliary loss keeps usage spread out.
- Communication. Experts sit on different devices, so routing means all-to-all traffic that can dominate wall-clock time.
- Instability. Sparse routing is noisy early in training and needs care to converge.
Why it's everywhere now
The current generation of open models has shown these problems are tractable at scale. Sparsity turned from a research curiosity into a default tool: it's how you buy capability without paying for it on every token. Expect the interesting questions to move to routing quality — sending each token to the expert that actually helps it.
