I thought I found a great deal on an H100.

~$2.50/hour. Way cheaper than what I’d seen elsewhere.

On paper, it looked like a no-brainer.

It wasn’t.

The mistake I made

Like most people, I compared GPU providers based on:

hourly price

That’s how every pricing page is structured.

So naturally, that’s how we evaluate them.

But after actually running workloads, it became obvious:

the hourly rate is one of the least important numbers.

What actually matters: cost per useful compute

The real question isn’t:

“How much does this GPU cost per hour?”

It’s:

“How much does it cost to get the result I want?”

Training run. Inference throughput. Completed job.

Once you look at it that way, things change fast.

Where the extra cost comes from

Here are the biggest ones I’ve seen:

1. Idle GPUs (this adds up fast)

GPUs are rarely fully utilized.

jobs wait on data

pipelines stall

you overprovision “just in case”

If your GPU is sitting idle 30–40% of the time, your “cheap” instance isn’t cheap anymore.

2.…