I used one RTX PRO 6000 Blackwell 96 GB per model to run three copies of a local LLM as coding workers at the same time (llama.cpp b11803, CUDA, Q8_0 GGUF, --parallel 3 , 64K context per slot). A cloud model planned, the GPU did all the code writing. Same task, three new MoE models.

Metric Flint 40B-A2.5B Ornith 1.5 35B-A3B Nemotron 3.5 Lightning 30B-A3B

Decode, 1 request 244 t/s 143 t/s 295 t/s

Decode per request, 3 at once 135 t/s 72 t/s 151 t/s

Prefill 5,230 t/s 2,950 t/s 7,670 t/s

Max context seen 54K 29K 57K

Wall time for the task 7:36 9:04 30:32

Hardware takeaways:

Going from 1 to 3 concurrent requests cut per-request decode to roughly 50-55% on all three, so total throughput only rises ~1.5-1.7x. Still worth it for parallel agents, just not 3x.

Nemotron was fastest on the card in every row and finished 4x slower, because it generated ~5x more tokens redoing failed edits. On agent work the GPU number is only half the story.

Caveats: one run per model.…