I wanted to see how far you can get making a pretrained symbolic music model "listen" to a live player without training it. Turns out pretty far.

Setup: I play a MIDI keyboard, the Anticipatory Music Transformer (Thickstun et al., music-medium-800k) plays a second piano. While I play I track my key, the chord I'm holding and my tempo, and add biases to the logits before sampling. Out-of-key notes get pushed down, chord tones and tensions get pushed up, the keys right around my hands get pushed down so it doesn't step on me, and when I stop it gets pulled up to where a melody would be. There's also a small boost for continuing the intervals of my last phrase, which makes it sound like it's answering.

It samples about a second ahead, and when I change chord I throw away what it had planned and resample. Runs on a Jetson Orin Nano.

Video + code: https://github.com/RiccardoCataldi/duetto

The offsets are all hand-tuned by ear, which feels a bit dumb.…