NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit

Serving large language models at scale is fundamentally a memory problem. During autoregressive decoding, every generated token requires reading the entire weight matrix and the accumulated KV cache from GPU high-bandwidth memory (HBM). On an H100, that bandwidth ceiling is around 3.35 TB/s — fast, but finite. As context windows grow and batch sizes increase, the gap between what the GPU can compute and what it can actually move through memory widens.

Lossless weight compression is one lever practitioners can pull to ease this bottleneck. The idea is simple: store weights in a smaller format, decompress on the fly during inference, and net a bandwidth gain. The catch is that decompression itself must be cheap enough not to eat the savings.…