Menu

KTransformers Review: Trading GPU VRAM for CPU RAM and NUMA Complexity
📰

KTransformers Review: Trading GPU VRAM for CPU RAM and NUMA Complexity

DEV Community·Jangwook Kim·3 days ago
#JG6rJoTW
#dev#memory#output#model#socket
Reading 0:00
15s threshold

Why We Are Evaluating KTransformers

We put KTransformers on our evaluation list because “buy enough GPUs to hold every expert” is an expensive answer to a sparse-model problem. A large mixture-of-experts model activates only a subset of its experts for each token, but its inactive weights still need somewhere to live.

Published DeepSeek-V3 Memory Requirements Single socket DRAM 382G DRAM Dual socket DRAM 1T DRAM V0.3 preview DRAM 644 GB DRAM Single-socket VRAM minimum 14 GB VRAM

The published configurations trade GPU memory for substantial CPU RAM; these capacity figures are not interchangeable measured memory peaks.

KTransformers makes a different allocation: keep routed experts in CPU memory and execute their computation on the CPU, while GPU-resident components handle attention and other compute-intensive work. The SGLang integration can also keep selected hot experts on GPUs.

That is not ordinary layer offloading.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.