Testing Long-Horizon Coherence in MusicGen: A Real-Data Mech Interp Pipeline
I have been working on a mechanistic interpretability experiment for music generation models.
The big motivating question is simple:
Do autoregressive music models have internal features that track long-horizon musical structure, or are they mostly stitching together locally plausible audio?
More concretely, I want to test whether a model like MusicGen has residual-stream or sparse-autoencoder features that say something like: "this motif should come back later," "this tension should resolve," or "this section is setting up a later recurrence."
That sounds exciting, but let me be clear upfront:
This post does not claim that I found foresight circuits.
What I have so far is a real-data pipeline, a benchmark slice, cached activations, recurrence proposals, and a set of artifacts that make the next causal experiments possible.…