Language models read all their weights from memory once for every word they write. I tested whether a second, guessed word could ride along in the same read for almost no extra cost. The full model checks the guess, and a wrong guess is thrown away, so the output never changes.
The ride-along part works. Running a second word through each layer added only a few percent to the time on a MacBook Air M4 (16/256).
The guessing part does not. The guess comes from the first few layers, and those layers only get the next word right about a third of the time. Almost all of those right guesses are commas and small words like "the" and "and". In the end the whole thing ran about 14% slower than normal decoding.
The best speedup I found on this laptop came from an existing tool, llama.cpp with a small helper model that drafts words ahead. It ran about 20% faster than normal, and about 30% faster on code.
I stopped at a checkpoint I had set before starting.…