Menu

Prefix tuning changes no weights: 128 numbers steering a frozen model
πŸ“°

Prefix tuning changes no weights: 128 numbers steering a frozen model

DEV Community: deeplearningΒ·Devanshu BiswasΒ·6 days ago
#yBlO667l
#devtodeeplearning#education#devto#dev#prefix
Reading 0:00
15s threshold

LoRA computes W_eff = W + BA . It changes a weight matrix by a low-rank delta, and at serve time you can add the delta in and forget it was ever separate.

Prefix tuning never forms a delta. It leaves every weight bit-for-bit where it was and prepends P learned key rows and P learned value rows to the attention of every layer. Two concat lines, and the softmax does the rest.

That difference is the whole subject, and it is easy to lose in the phrase "parameter-efficient fine-tuning", which gets applied to both.

πŸ‘‰ Live, with the training run executing in your browser: https://dev48.in/dl/day83-prefix-tuning.html

πŸ”΄ The model is a toy and the numbers are a property of it

The page builds a miniature decoder from a fixed seed: d = 8 , 2 attention layers, 6 tokens per sequence, a 12-token vocabulary, 2 classes. 624 numbers in total. Re-running reproduces it exactly.

What that costs: every point estimate below is a property of this model, and would move on a real one.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.