Menu

How to fine-tune an embedding model with LoRA adapters
📰

How to fine-tune an embedding model with LoRA adapters

DEV Community: machinelearning·Ethan Steininger·12 days ago
#uw0uudlb
Reading 0:00
15s threshold

To fine-tune an embedding model with LoRA, freeze the base encoder, attach a small LoRA adapter to its linear layers, and train only the adapter with a contrastive loss on (query, relevant passage) pairs from your own domain. With Sentence Transformers this is a few lines: build a LoraConfig with task_type=FEATURE_EXTRACTION , call model.add_adapter() , and train with MultipleNegativesRankingLoss . A few hundred to a few thousand clean pairs is usually enough. The adapter comes out at a few megabytes. Afterwards you must re-embed your whole corpus, because the fine-tuned model's vectors are not comparable with the old ones.

Why LoRA fits embedding models

Full fine-tuning updates every weight and leaves you with a second full copy of the model. LoRA freezes the weight matrix W and learns a low-rank correction B·A on top of it, scaled by alpha / r . Only A and B train.

For an encoder that matters in three ways:

Small files.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.