Menu

Full fine-tune vs LoRA vs QLoRA on one L40S: Qwen2.5-1.5B, memory, speed and held-out loss, measured
📰

Full fine-tune vs LoRA vs QLoRA on one L40S: Qwen2.5-1.5B, memory, speed and held-out loss, measured

DEV Community: pytorch·Runcrate·18 days ago
#gRrtYYFi
#devtopytorch#devto#dev#model#lora
Reading 0:00
15s threshold

"Use LoRA, it needs far less memory" and "use QLoRA if LoRA doesn't fit" are the standard advice for fine-tuning on one GPU. I wanted to see what each one costs on a small model, where the weights are not the problem, so I rented an L40S (48 GB) on Runcrate today (2026-10-28) and fine-tuned Qwen2.5-1.5B-Instruct three ways on the same data for the same number of steps. Two machines, both deleted when the results were on disk.

Short version: full fine-tuning needed 36.9 GiB at micro-batch 4 and ran out of memory at micro-batch 8. LoRA needed 14.5 GiB, trained 1.4x faster and reached a slightly lower held-out loss. QLoRA was slower than both and used more memory than LoRA, not less.

Setup

GPU: one L40S (48 GB, 44.4 GiB usable), Kansas City, US, $1.14 per hour on demand at launch time.

Model: Qwen/Qwen2.5-1.5B-Instruct (1.54 billion parameters).

Data: yahma/alpaca-cleaned, shuffled with a fixed seed: 4,000 examples for training (753,486 tokens), the next 300 held out.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.