"Use LoRA, it needs far less memory" and "use QLoRA if LoRA doesn't fit" are the standard advice for fine-tuning on one GPU. I wanted to see what each one costs on a small model, where the weights are not the problem, so I rented an L40S (48 GB) on Runcrate today (2026-10-28) and fine-tuned Qwen2.5-1.5B-Instruct three ways on the same data for the same number of steps. Two machines, both deleted when the results were on disk.
Short version: full fine-tuning needed 36.9 GiB at micro-batch 4 and ran out of memory at micro-batch 8. LoRA needed 14.5 GiB, trained 1.4x faster and reached a slightly lower held-out loss. QLoRA was slower than both and used more memory than LoRA, not less.
Setup
GPU: one L40S (48 GB, 44.4 GiB usable), Kansas City, US, $1.14 per hour on demand at launch time.
Model: Qwen/Qwen2.5-1.5B-Instruct (1.54 billion parameters).
Data: yahma/alpaca-cleaned, shuffled with a fixed seed: 4,000 examples for training (753,486 tokens), the next 300 held out.…