Menu

Layer-hot decoding: an approach to check a guessed token while each layer's weights are already loaded on an Apple M4. It ran slow. [R]

Reading 0:00
15s threshold

Language models read all their weights from memory once for every word they write. I tested whether a second, guessed word could ride along in the same read for almost no extra cost. The full model checks the guess, and a wrong guess is thrown away, so the output never changes.

The ride-along part works. Running a second word through each layer added only a few percent to the time on a MacBook Air M4 (16/256).

The guessing part does not. The guess comes from the first few layers, and those layers only get the next word right about a third of the time. Almost all of those right guesses are commas and small words like "the" and "and". In the end the whole thing ran about 14% slower than normal decoding.

The best speedup I found on this laptop came from an existing tool, llama.cpp with a small helper model that drafts words ahead. It ran about 20% faster than normal, and about 30% faster on code.

I stopped at a checkpoint I had set before starting.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.