Hey developers!

My name is Ayushman Singh Rathore, and over the past several months I have been building VeloGrad (v1.0.0), a native-first machine learning and autoregressive inference runtime written from scratch in modern C++20, Rust, and CUDA.

GitHub Repository: https://github.com/IDKLETSCODESOMETHIN/VeloGrad

Why I Built It: PyTorch is great for research, but running modern long-context LLMs in production usually requires duct-taping multiple heavy frameworks together (vLLM, DeepSpeed, TensorRT-LLM). This carries Python GIL stalls, massive multi-gigabyte dependencies, and severe memory fragmentation.

I wanted to explore how far we could push runtime efficiency by designing an engine from bare-metal primitives:

Static Arena Allocator: Instead of dynamic heap allocations during inference, memory is managed via a contiguous arena aligned to 64-byte hardware cache lines (alignas(64)), eliminating GC stalls and latency jitter.

Virtual Paged Attention: We break the KV-cache into discrete physical pages…