Menu

Human Preference Tuning for Small VLMs: SmolVLM2 + DPO in Transformers 5.12.1
📰

Human Preference Tuning for Small VLMs: SmolVLM2 + DPO in Transformers 5.12.1

DEV Community: computervision·Harpreet Sahota·9 days ago
#FmtMJ9xD
Reading 0:00
15s threshold

Human Preference Tuning for Small VLMs: SmolVLM2 + DPO in Transformers 5.12.1

TL;DR

Direct Preference Optimization (DPO) lets you align a vision-language model from preferred and rejected answers, without training a separate reward model.

The older HuggingFaceTB/SmolVLM-Instruct example is worth updating to HuggingFaceTB/SmolVLM2-2.2B-Instruct , which is newer, Apache 2.0, and documented with AutoModelForImageTextToText .

For current TRL vision-language DPO, keep the dataset in images , prompt , chosen , and rejected form, pass a processor through processing_class , and set max_length=None so image tokens are not truncated away.

Abstract

Preference tuning is one of the most practical ways to make a vision-language model behave less like a generic captioning engine and more like the assistant you actually want. Instead of asking the model to imitate one answer, DPO asks it to prefer one answer over another for the same image and prompt.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.