Menu

Inference cost at scale with napkin math

InJuly·5 months ago
#ocIRDD39
#gpu#ai#math#memory#cost
Reading 0:00
15s threshold

index about blog work now

If you serve AI models as a part of your product stack, you must know what kind of load your GPU cluster tops out at.

With some rudimentary knowledge about your hardware and model architecture, we can work out the dollar cost-per-user on the back of a napkin 1 .

Resources on a single GPU

Cost of a Matrix Multiplication

An Overview of Language Models

Attention in Greater Detail

Reducing Compute with KV-Cache

How much does a token cost?

How many users can you serve realistically?

Optimizing for hundreds of users on a GPU

Tokens Per Second

Dollar cost per user

Resources on a single GPU

On a GPU's spec-sheet you will find these metrics:

Peak throughput: Number of floating-point operations per second.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.