A 0.5B-parameter chat model, 4-bit quantized, is about 490 MB. That fits on any recent phone, and llama.cpp runs it on the CPU with no GPU, no server and no API key. The hard part is not the inference. It's making it feel like a normal Kotlin API inside a normal Android app.
This post is about the three problems that took most of the time when I built llama-kotlin , a small Kotlin library over llama.cpp, and a fully offline chat app on top of it.
The API I wanted
val model = LlamaModel . load ( File ( filesDir , "qwen2.5-0.5b-instruct-q4_k_m.gguf" ))
model . newSession (). use { session -> session . chat ( listOf ( ChatMessage . system ( "You are a concise assistant." ), ChatMessage . user ( "Explain coroutines in one sentence." ), ) ). collect { piece -> print ( piece ) }
println ( "\n${session.lastStats?.tokensPerSecond} tokens/s" ) }
Enter fullscreen mode
Exit fullscreen mode
chat() returns a cold Flow<String> .…