Ai2th
Posted on Nov 15
Edited on Nov 15
Most AI features in apps are a thin wrapper around a cloud API. Miktam Bible's study assistant is different: the language model runs on the phone , and every answer is grounded in verses pulled from the offline database.
🧠 The model
The assistant runs SmolLM2-135M , a small model quantized to 4 bits in GGUF format (about a 105 MB one-time download), through llama.rn , the React Native binding for llama.cpp. The settings are chosen for phones:
context = await initLlama ( { model : nativePath , n_ctx : 2048 , // Mobile-safe context window n_batch : 256 , // Mobile-safe batch size n_threads : 4 , // CPU threads for inference n_gpu_layers : 0 , // 0 = CPU only use_mlock : false , use_mmap : true , // Memory-map the model file } ) ;
Enter fullscreen mode
Exit fullscreen mode
CPU-only with a memory-mapped model file keeps it working across the wide range of Android hardware.…