Book: LLM Observability Pocket Guide: Picking the Right Tracing & Evals Tools for Your Team

Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go

My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools

Me: xgabriel.com | GitHub

You are running a 70B model on two H100s for an internal coding

assistant. The product team wants the median response under two

seconds. The model is sitting at 4.1s. You already swapped to

FP8 quantization. You already turned on continuous batching. The

GPU memory is comfortable. The bottleneck is the one thing

quantization does not fix: every output token is one full forward

pass through 70 billion parameters, in series, one after the

next. A 200-token answer is 200 of those passes back to back.

Speculative decoding is the trick that breaks the series.

What it actually does

The idea, from Leviathan et al., 2023 ,

is to split the work in two.…