Menu

#Llama

131 posts

Feed·
20 of 131 posts
Developer's llama.cpp Fork Makes Prompt Lookup Drafting Up to 42x Faster
🖼️
0Read More

Developer's llama.cpp Fork Makes Prompt Lookup Drafting Up to 42x Faster

#webpronews#news#cache#llama#drafting

Hayder Tirmazi rebuilt the n-gram caches behind llama.cpp's prompt lookup decoding, cutting drafting time per token by up to 42x and memory use by up to 2.6x. The changes sit in his fork and haven't been merged upstream.

15s
Compress Gemma 4 by 3x with 2x Speedup: Complete Air-26B-A4B Multimodal Deployment Guide on Consumer GPUs
📰
0Read More

Compress Gemma 4 by 3x with 2x Speedup: Complete Air-26B-A4B Multimodal Deployment Guide on Consumer GPUs

DEV Community: deeplearning·Rich Ng·3 days ago
#nTHhwufM
#devtodeeplearning#education#devto#dev#llama

1. The Multimodal Efficiency Paradox: Why Massive Models Fail in Production The artificial...

15s
GGML vs llama.cpp v1.0: Build an Inference Engine or Use One?
📰
0Read More

GGML vs llama.cpp v1.0: Build an Inference Engine or Use One?

DEV Community: cpp·jamilxt·4 days ago
#1O0iyEIc
#devtocpp#devto#dev#model#llama

A model file is not an inference product. Someone still has to load its tensors, schedule operations,...

15s
llama.cpp and ggml ship v1.0.0 — semantic versioning comes to the local AI stack
🖼️
0Read More

llama.cpp and ggml ship v1.0.0 — semantic versioning comes to the local AI stack

DEV Community: cpp·News·5 days ago
#djUcyUwF
#devtocpp#devto#dev#ggml#release

📅 November 13, 2026 • Source: Hugging Face ggml-org has shipped llama.cpp v1.0.0 together with...

15s
Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
📰
0Read More

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

DEV Community·xbill·10 days ago
#T30kREUG
#dev#vllm#sglang#drafter#llama

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 8 GB RTX 3050, measured with the same prompts, the same load sweep and the same GSM8K and BFCL questions.…

15s
Serving Sage-A1.5 on an RTX 4090: one broken flag, one queue, and a self-check mode to measure
📰
0Read More

Serving Sage-A1.5 on an RTX 4090: one broken flag, one queue, and a self-check mode to measure

#devcommunity#dev#model#card#llama

Sage-A1.5 ran at about 136 tokens/s on one 24 GB card. The model card's llama.cpp flag fails, and Verification Mode doubled tokens on easy questions.

15s
Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4
📰
0Read More

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4

DEV Community: aws·xbill·12 days ago
#rJLJU2lc
#devtoaws#devto#dev#vllm#drafter

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one g6.4xlarge, measured with the same prompts, the same 1-to-64-request sweep and the same GSM8K and BFCL questions.…

15s
Llama vs Qwen vs DeepSeek: How Much VRAM Do You Need to Self-Host?
📰
0Read More

Llama vs Qwen vs DeepSeek: How Much VRAM Do You Need to Self-Host?

DEV Community: ai·Thea Lauren·13 days ago
#kWfWxuSp
#devtoai#devto#dev#model#vram

A practical VRAM and GPU sizing guide for hosting Llama 3, Qwen 3 and DeepSeek V3/R1 on bare-metal servers.

15s
Running an LLM fully offline on Android with Kotlin coroutines
📰
0Read More

Running an LLM fully offline on Android with Kotlin coroutines

DEV Community: ai·Esdras Souza·16 days ago
#x7AetUjn
#devtoai#devto#dev#session#llama

How I wrapped llama.cpp in a small, coroutine-first Kotlin API, and the three problems that took the most time: threads, cancellation and broken emoji.

15s