Menu

LoRA on spot RTX 4090s: what happens when the host dies mid-run.
πŸ“°

LoRA on spot RTX 4090s: what happens when the host dies mid-run.

DEV Community: machinelearningΒ·Andy WhiteΒ·7 days ago
#PLnJLILg
#devtomachinelearning#education#devto#dev#machine
Reading 0:00
15s threshold

Community GPU marketplaces like Vast.ai rent an RTX 4090 for well under a dollar an hour. For a lot of fine-tuning

work that is the cheapest compute you can get. The catch is that the machine belongs to someone else: hosts reboot,

lose their network, get outbid, or simply disappear in the middle of your run.

I wanted to know two things: how much cheaper a real fine-tune is on such a machine, and what it takes for that

fine-tune to survive the host disappearing.

The job

QLoRA fine-tune of Qwen2.5-7B-Instruct (4-bit), yahma/alpaca-cleaned, 300 steps, checkpoint every 50 steps.

Public image pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime , dependencies installed at start.

One RTX 4090, host reliability β‰₯ 0.98, network β‰₯ 500 Mbit/s.

What it cost

The number below is taken from the Vast.ai invoice for one run without failures, not from "price Γ— time".

Time on the machine 0.44 h (about 19 min of training, the rest is start-up and the model download)

Billed by Vast.ai β‰ˆ $0.19

Same 0.44 h on one A100 at…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.