Menu

Kindi – fast local inference on low-cost GPUs, and the exact-match rule that makes its numbers trustworthy
📰

Kindi – fast local inference on low-cost GPUs, and the exact-match rule that makes its numbers trustworthy

DEV Community: machinelearning·Private Local AI·about 10 hours ago
#tQj3N0gz
Reading 0:00
15s threshold

Kindi (AGPL-3.0 with a commercial-license path, github.com/ahtx/kindi) is Amir Husain's inference engine for cheap and old hardware, and its v0.6 release does something no engine I have covered here has done: it made being correct a hard constraint, then benchmarked itself against llama.cpp on the same card, with the same model file and the same context, line by line. When the numbers say "20 times faster than llama.cpp on a 1 GB Quadro K420", they come attached to a kindi verify run that grades the engine token-for-token against llama.cpp's reference output — 40/40 tokens matching, four prompts, before any speed claim is allowed to exist.

That design order is the whole story for me: this blog has been saying for months that a speed claim without an agreement metric beside it is a claim I cannot use ( Xolotl's notes are the clearest example).…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.