Menu

Before you rent the GPUs, check the map: I built a CLI that picks your LLM serving engine
πŸ“°

Before you rent the GPUs, check the map: I built a CLI that picks your LLM serving engine

DEV Community: pythonΒ·TECHDEVΒ·16 days ago
#jjYUegG9
#devtopython#devto#dev#servefit#model
Reading 0:00
15s threshold

DeepInfer collected ~19k stars in a single week (GitHub Trending scan 2026-10-28). The headlines say "production inference engine". My weekend said: it serves exactly one model (DeepSeek-V4.1-Flash), needs SM100+ or Ascend950 silicon, and runs offline-only with no KV cache system, no PD disaggregation, no load balancing. Checked against the upstream repo 2026-11-01.

That is not a knock. That is the gap a tool should close: before you rent GPUs or rewire your stack, you need the map β€” what you actually have, what the model actually costs in memory, and which engines are actually viable on that combination.

So I built servefit : a zero-dependency Python CLI that inventories your hardware, does the memory math, checks a dated engine-support matrix, and recommends an engine with reasons. Two minutes, no account, and the offline path needs no network at all.

What it does

servefit inventory β€” parses nvidia-smi (GPUs, VRAM, compute capability with name fallback), AMD via rocm-smi , CPU/RAM.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.