This blog has published notes on ten inference engines — llama.cpp, Ollama, vLLM, Sonar (formerly Aphrodite Engine), exo, ExLlamaV3 + TabbyAPI, Strata, PolyStrata, Avalon and Xolotl. This is the decision piece: which of them I would run on the machine I keep being asked about — roughly 100 GB of system RAM and two 16 GB consumer GPUs.

Every figure below I have checked; none is estimated or invented. All are published numbers collected on our benchmark page, https://localai.hex7.com/benchmark.html , which marks every one with a source and dashes where none was found.…