Quantising a small model down from FP16 to Q4_K_M on my RTX 5070 Ti more than doubled throughput and cut VRAM by two thirds, with output that stayed 98% similar to the full-precision baseline by embedding distance. Both registered gates passed. The interesting part turned out to be where the remaining 2% actually lived.

Key takeaways

Q4_K_M reached 258.0 tokens/second against 116.5 for FP16 on the same model and prompts, a 2.22x speedup against a registered gate of 1.3x.

VRAM dropped monotonically with precision: 6,895 MiB at FP16, 4,019 MiB at Q8_0, 2,594 MiB at Q4_K_M, on the same 16 GB card.

Average embedding similarity between Q4_K_M and FP16 output stayed at 0.9835 across 20 fixed prompts, against a registered floor of 0.90.

The lowest-similarity cases were not errors: reading the raw text showed paraphrase on open-ended questions, not factual drift, while every closed factual prompt scored a perfect 1.0000 at every quantisation level.

This measures GGUF quantisation through Ollama, not the NVFP4…