Menu

We built a P2P AI inference network that runs on Android phones — here's what we learned

DEV Community: android·Vishal Sharma·7 months ago
#ZQGTPAT8
#dev#code#strong#android#model
Long article (873 words) — no AI summary yet.
Reading 0:00
15s threshold

I'm one of the founders of Nataris. We built a P2P inference marketplace where Android phones run open-weight AI models and serve developer API requests. Phone owners earn per token. Developers get an OpenAI-compatible API.

This is a writeup of the genuinely hard technical problems we ran into — not a product pitch.

The idea

There are currently two ways to run AI inference: on your own machine locally, or through a big company's datacenter. We wanted to build the third option — inference that runs on real people's Android phones, accessible via a standard API, where the phone owners get paid.

The privacy properties fall out naturally: inference never touches a server we own. No prompt logging. No content filtering. No model training on queries.

Problem 1: llama.cpp OOM crashes on mobile

llama.cpp reads context_length from GGUF metadata and allocates the full KV cache upfront.

Llama 3.2 1B ships with a 131K context window — that's ~4GB of KV cache on a phone with maybe 2GB free RAM.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.