Prompt lookup decoding in llama.cpp just got a lot cheaper to run, at least in one developer’s fork. In a blog post published Sept. 26 , Hayder Tirmazi said a set of data-structure fixes makes drafting for prompt lookup “up to 42x faster while using up to 2.6x less memory,” building largely on the work of Daniel Lemire and Martin Ankerl.
Prompt lookup decoding, also called n-gram speculation, is a form of speculative decoding that swaps the usual small draft model for an n-gram model. The engine checks the most recent tokens against cached n-grams and proposes whatever token usually came next. It needs no extra model, and it does best when a model’s output repeats its input, as in code rewrites. llama.cpp keeps three such caches: a context cache for the current tokens, a dynamic cache built from earlier runs, and a static cache built from a text corpus with llama-lookup-create.
Where the speedup came from
Tirmazi’s first change was close to a bug fix.…