index about blog work now
If you serve AI models as a part of your product stack, you must know what kind of load your GPU cluster tops out at.
With some rudimentary knowledge about your hardware and model architecture, we can work out the dollar cost-per-user on the back of a napkin 1 .
Resources on a single GPU
Cost of a Matrix Multiplication
An Overview of Language Models
Attention in Greater Detail
Reducing Compute with KV-Cache
How much does a token cost?
How many users can you serve realistically?
Optimizing for hundreds of users on a GPU
Tokens Per Second
Dollar cost per user
Resources on a single GPU
On a GPU's spec-sheet you will find these metrics:
Peak throughput: Number of floating-point operations per second.…