Sparse Autoencoders: Why Forcing Sparsity Makes Neural Features Interpretable
A trained neural network knows a lot more than it can explain. Its activations contain information about syntax, facts, plans, and quirks of the training data, but that information is usually tangled. A single neuron may respond to several unrelated ideas, and a single idea may be spread across many neurons. Interpretability researchers call this superposition: when a model has more features to represent than it has dimensions to represent them in, it packs multiple features into overlapping directions.
Sparse autoencoders, often shortened to SAEs, are a practical response to that problem. They do not try to explain a model neuron by neuron. Instead, they learn a much larger dictionary of directions, and force each activation to be explained by only a few of them.…