The idea that the neural networks underpinning modern AI are “black boxes” almost impossible to understand has become received wisdom. But a provocative blog post from a researcher who played a key role in setting that narrative says these models may be far more interpretable than we thought.
The opacity of today’s leading AI models is becoming an increasingly pressing problem as the technology seeps into ever more areas of our lives. Recent incidents where agents from leading developers escaped their sandboxes to go on hacking sprees have only boosted the sense of urgency.
This has spurred interest in a research field known as mechanistic interpretability, which attempts to reverse engineer neural networks to learn what individual neurons and the circuits they create do.…