Menu

Running an LLM fully offline on Android with Kotlin coroutines
📰

Running an LLM fully offline on Android with Kotlin coroutines

DEV Community: ai·Esdras Souza·16 days ago
#x7AetUjn
#devtoai#devto#dev#session#llama
Reading 0:00
15s threshold

A 0.5B-parameter chat model, 4-bit quantized, is about 490 MB. That fits on any recent phone, and llama.cpp runs it on the CPU with no GPU, no server and no API key. The hard part is not the inference. It's making it feel like a normal Kotlin API inside a normal Android app.

This post is about the three problems that took most of the time when I built llama-kotlin , a small Kotlin library over llama.cpp, and a fully offline chat app on top of it.

The API I wanted

val model = LlamaModel . load ( File ( filesDir , "qwen2.5-0.5b-instruct-q4_k_m.gguf" ))

model . newSession (). use { session -> session . chat ( listOf ( ChatMessage . system ( "You are a concise assistant." ), ChatMessage . user ( "Explain coroutines in one sentence." ), ) ). collect { piece -> print ( piece ) }

println ( "\n${session.lastStats?.tokensPerSecond} tokens/s" ) }

Enter fullscreen mode

Exit fullscreen mode

chat() returns a cold Flow<String> .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.