Menu

#Vllm

76 posts

Feed·
20 of 76 posts
Validating GPU Platforms with Runtime AI Inventory via the NVIDIA DSX AI Factory
🖼️
0Read More

Validating GPU Platforms with Runtime AI Inventory via the NVIDIA DSX AI Factory

NVIDIA Technical Blog·Mark Chmarny·6 days ago
#h2V3utvM
#developer#aibom#aicr#deployment#vllm

As teams deploy and scale AI workloads using NVIDIA Dynamo, Kubeflow, or Slurm, tracking what actually runs requires visibility beyond the platform…

15s
Why vLLM and SGLang might not be the best option to run AI agents
🖼️
0Read More

Why vLLM and SGLang might not be the best option to run AI agents

DEV Community: rust·Nahuel Nucera·6 days ago
#4eu6yLdp
#devtorust#devto#dev#agent#vllm

I replayed 24 real coding-agent sessions against vLLM, SGLang and a small Rust engine I wrote. The general-purpose engines recomputed more than half of every prompt.

15s
Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
📰
0Read More

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

DEV Community·xbill·9 days ago
#T30kREUG
#dev#vllm#sglang#drafter#llama

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 8 GB RTX 3050, measured with the same prompts, the same load sweep and the same GSM8K and BFCL questions.…

15s
Zero-Downtime LLM Inference on Kubernetes Spot Instances (~70% GPU Cost Savings)
📰
0Read More

Zero-Downtime LLM Inference on Kubernetes Spot Instances (~70% GPU Cost Savings)

DEV Community: kubernetes·riyankhamdani·9 days ago
#bxBlncSD
#devtokubernetes#devto#dev#spot#node

Self-hosting open-source Large Language Models (LLMs) like Llama 3, DeepSeek, or Mistral on dedicated...

15s
Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4
📰
0Read More

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4

DEV Community: aws·xbill·11 days ago
#rJLJU2lc
#devtoaws#devto#dev#vllm#drafter

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one g6.4xlarge, measured with the same prompts, the same 1-to-64-request sweep and the same GSM8K and BFCL questions.…

15s
Gemma 4 E2B in JAX on One TPU v5e: bf16 or 4-Bit, Python or Rust, and How It Stands Against vLLM
📰
0Read More

Gemma 4 E2B in JAX on One TPU v5e: bf16 or 4-Bit, Python or Rust, and How It Stands Against vLLM

DEV Community: rust·xbill·12 days ago
#SrvR006u
#devtorust#devto#dev#engine#rust

Gemma 4 E2B served three ways on one Cloud TPU v5e chip: a pure-JAX engine in Python, the same graph written in Rust and compiled by the libtpu that JAX loads, and vLLM, all timed by one client.…

15s
Serving WeVisOne-VL-4B: two engines, two copies of the weights, one chess piece apart
📰
0Read More

Serving WeVisOne-VL-4B: two engines, two copies of the weights, one chess piece apart

DEV Community: computervision·InferenceKey·14 days ago
#sAMTYwQ5

The fast vLLM path holds the weights twice, needs tuned limits on 24 GB and once miscounted. The reference path ran every task on 16 GB.

15s
Running DiffusionGemma Locally on an RTX 5090 with vLLM: The One-Time Pioneer Cost of Bleeding-Edge Model Bleeding-Edge GPU
📰
0Read More

Running DiffusionGemma Locally on an RTX 5090 with vLLM: The One-Time Pioneer Cost of Bleeding-Edge Model Bleeding-Edge GPU

DEV Community: machinelearning·kohei_wada·22 days ago
#HPL1ARuU

I stood up Google's diffusion language model DiffusionGemma as a vLLM OpenAI-compatible server on my home RTX 5090 (Blackwell / 32GB), so Open WebUI could use it as just another model. Declarative, on-demand on NixOS.…

15s
vLLM KV Cache Math: Why My 24GB GPU Served Only 4 Requests at Once
📰
0Read More

vLLM KV Cache Math: Why My 24GB GPU Served Only 4 Requests at Once

DEV Community: python·jidonglab·25 days ago
#5GTMDwNl
#devtopython#devto#dev#cache#token

vLLM KV cache sizing explained: bytes per token, why a 24GB GPU runs 4 RAG requests at once, and the flags that raise concurrency.

15s
GitHub - ryanzhou/deepseek-v4-flash-mi300x
📰
382Read More

GitHub - ryanzhou/deepseek-v4-flash-mi300x

Hacker News·3 months ago
#wd1MZPnX
#github#vllm#prefill#mi300x#token

Contribute to ryanzhou/deepseek-v4-flash-mi300x development by creating an account on GitHub.

15s
0Read More

Why we write our own C and C++ engines

Hacker News·4 months ago
#HtyoLCQQ
#localai#python#against#parity#vllm

LocalAI is the open source AI engine. Run any model, LLMs, vision, voice, image and video, on any hardware. No GPU required.

15s
amd-strix-halo-vllm-toolboxes/rdma_cluster/setup_guide.md at main · kyuz0/amd-strix-halo-vllm-toolboxes
🖼️
232Read More

amd-strix-halo-vllm-toolboxes/rdma_cluster/setup_guide.md at main · kyuz0/amd-strix-halo-vllm-toolboxes

Hacker News·5 months ago
#hohDSTA5
#github#rdma#cluster#sudo#node

Contribute to kyuz0/amd-strix-halo-vllm-toolboxes development by creating an account on GitHub.

15s
GitHub - huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
🖼️
143Read More

GitHub - huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

Hacker News·6 months ago
#1P7kvTiP
#github#kvarn#cache#vllm#capacity

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag. - huawei-csl/KVarN

15s