Menu

I built HexCore — an ultra-lightweight, zero-dependency Paged & Quantized (FP16 / INT4 / INT2) KV-Cache Engine written in pure C++ and CUDA.
📰

I built HexCore — an ultra-lightweight, zero-dependency Paged & Quantized (FP16 / INT4 / INT2) KV-Cache Engine written in pure C++ and CUDA.

DEV Community: cpp·Rasuljanov Muhammadali·about 10 hours ago
#7YZlxZit
#devtocpp#devto#dev#cpp#llm
Reading 0:00
15s threshold

While production serving stacks like vLLM @FlashInfer SGLang, and LMDEploy are powerhouse frameworks, they carry massive dependency trees and architectural overhead. My goal with HexCore was to build a clean, auditable, standalone C++/CUDA memory engine designed for maximum VRAM compression and minimal footprint.

I just completed full hardware verification on an NVIDIA Tesla T4 (sm_75) using compute-sanitizer:

✅ Numerical Correctness (21/21 checks passed):

FP16 Decode & Prefill max error vs. CPU reference: ~4.8e-4

INT4 & INT2 Quantized Decode/Prefill max error: < 1.0e-3

Sentinel block (-1) & sequence bounds guards verified on-device

✅ Zero Memory & Concurrency Bugs:

compute-sanitizer --tool memcheck: 0 errors

compute-sanitizer --tool racecheck: 0 hazards, 0 warnings

Host allocator & Prometheus exporter hardened with ASan + UBSan (-Wall -Wextra -Werror -pedantic)

📊 Current Baseline & Next Optimization (8k context, Tesla T4):

batch=1: 5.65 ms (5.9 GB/s)

batch=8: 6.34 ms (42.3 GB/s)

Currently…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.