LoRA computes W_eff = W + BA . It changes a weight matrix by a low-rank delta, and at serve time you can add the delta in and forget it was ever separate.
Prefix tuning never forms a delta. It leaves every weight bit-for-bit where it was and prepends P learned key rows and P learned value rows to the attention of every layer. Two concat lines, and the softmax does the rest.
That difference is the whole subject, and it is easy to lose in the phrase "parameter-efficient fine-tuning", which gets applied to both.
π Live, with the training run executing in your browser: https://dev48.in/dl/day83-prefix-tuning.html
π΄ The model is a toy and the numbers are a property of it
The page builds a miniature decoder from a fixed seed: d = 8 , 2 attention layers, 6 tokens per sequence, a 12-token vocabulary, 2 classes. 624 numbers in total. Re-running reproduces it exactly.
What that costs: every point estimate below is a property of this model, and would move on a real one.β¦