Menu

WiCi Infer Runs 800B-Parameter AI Models on a Laptop by Streaming Them From the SSD
📰

WiCi Infer Runs 800B-Parameter AI Models on a Laptop by Streaming Them From the SSD

HackerNoon·sarahevans·about 5 hours ago
#qQMnhTVa
#hackernoon#technology#wici#memory#model
Long article (877 words) — no AI summary yet.
Reading 0:00
15s threshold

WiCi , the systems research lab behind the WiCi One wireless GPU, has released WiCi Infer, a free desktop app built to run large local LLMs on a personal computer. It runs 800B-parameter models on Macs and Windows PCs that can hold only a fraction of the model in memory. Until now, running a model that size has meant owning more memory than any laptop ships with.

The app does not load the full model into RAM or GPU memory before it answers. It keeps the weights on the computer's SSD and pulls in the parts each step needs.

WiCi Infer is built on InferSSD, the open-source inference engine the company recently released. Both make the same argument: frontier AI should run on hardware people already own, without a data center behind it.

How Does WiCi Infer Run Large LLMs Locally With Limited Memory?

WiCi Infer stores the model on the SSD and moves only the active portion into memory as text is generated.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.