Beyond Python: Benchmarking Modern SLMs (Gemma 4, Qwen 3 & Phi-4) on CPU with Pure Rust & Candle

When I first shared my experiment on running Microsoft’s Phi-3 on a standard consumer CPU using pure Rust and Hugging Face’s Candle framework, the mission was straightforward: prove that we don’t need bloated Python runtimes, complex CUDA setups, or 5GB+ Docker images just to run local AI inference.

Fast forward to today, and the Small Language Model (SLM) landscape has exploded. With the release of edge-optimized powerhouses like Google’s Gemma 4 (E4B) , Alibaba’s Qwen 3 / 3.6 (4B) , and Microsoft’s Phi-4-mini (3.8B) , running high-reasoning AI directly on standard CPU hardware is no longer just a weekend hack—it is a viable production architecture.

In this guide, we will upgrade our pure-Rust inference pipeline to build a multi-model CLI engine capable of running the latest quantized GGUF models on a standard CPU, complete with real-world benchmarks on RAM usage, cold-start latency, and token generation…