Menu

How much does an adapter forget?
📰

How much does an adapter forget?

DEV Community: deeplearning·Ivan Zdravkov·15 days ago
#zaheTLRB
Reading 0:00
15s threshold

Less than expected, and the honest story is why that answer took three extra checks to trust. Across a 3x3 grid of LoRA rank and training steps on Qwen2.5-3B-Instruct, general-task retention never once dropped below the unadapted baseline.

Key takeaways

The registered kill-gate for H1 (forgetting grows with adapter rank and training steps) did not fire: retention rose 9.1 percentage points between the smallest and largest configuration, the opposite of the hypothesis.

Every one of nine grid cells showed general-task retention at or above the unmodified baseline, a pattern suspicious enough to investigate before trusting it.

Two checks ruled out ordinary bugs: the raw generated text showed no parsing artifact, and reloading the identical unmodified model twice produced zero differences in its answers.

The real gap was a missing seed control in the original grid.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.