I'm one of the founders of Nataris. We built a P2P inference marketplace where Android phones run open-weight AI models and serve developer API requests. Phone owners earn per token. Developers get an OpenAI-compatible API.
This is a writeup of the genuinely hard technical problems we ran into — not a product pitch.
The idea
There are currently two ways to run AI inference: on your own machine locally, or through a big company's datacenter. We wanted to build the third option — inference that runs on real people's Android phones, accessible via a standard API, where the phone owners get paid.
The privacy properties fall out naturally: inference never touches a server we own. No prompt logging. No content filtering. No model training on queries.
Problem 1: llama.cpp OOM crashes on mobile
llama.cpp reads context_length from GGUF metadata and allocates the full KV cache upfront.
Llama 3.2 1B ships with a 131K context window — that's ~4GB of KV cache on a phone with maybe 2GB free RAM.…