Menu

GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.
📰
0

GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

Hacker News·Hacker News·about 2 months ago
#i9E3YRcs
#github#container#waste#cache#token#engine
Reading 0:00
15s threshold

WASTE — Weight-Aware Streaming Tensor Engine Kimi K3 — 2.78 trillion parameters — running on a consumer laptop. $ waste run ~/models/k3.waste 'What is the capital of Italy?' waste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB) The capital of Italy is **Rome**. [16 tokens, 31.09 s, 0.51 tok/s | experts 3357 hit / 20195 miss = 14%] WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache. Its current proof point is the complete open-weights Kimi K3 model: 2.78 trillion parameters, converted into a 982 GiB container and running on a 64 GB MacBook Pro at 0.49–0.54 tokens per second. This is not a distilled, pruned, or reduced variant .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More