Menu

UMM-JEPA: Predicting What the Model Understands, Not What the Decoder Wants
πŸ“°

UMM-JEPA: Predicting What the Model Understands, Not What the Decoder Wants

DEV Community: deeplearningΒ·Siddharth Singh BhadauriaΒ·about 18 hours ago
#f2ofIHmw
#devtodeeplearning#education#devto#dev#scale
Reading 0:00
15s threshold

Unified multimodal models stop fighting themselves when the generation objective targets the same continuous representation that understanding already uses and UMM-JEPA does this through masked, coarse-to-fine prediction of those features.

The Real Problem / Why Naive Approaches Fail

The pitch for a unified multimodal model is simple. One backbone reads images, writes images and answers questions about them. You expect the two skills to help each other: a model that can draw a dog should understand dogs better.

That isn't what usually happens. Generative training often adds little or nothing to understanding. Teams that bolt an image-generation objective onto a vision-language model frequently find their VQA and captioning numbers flat or slightly worse, even though the generation samples look good.

The UMM-JEPA abstract gives a plausible diagnosis. The two tasks optimize different representation spaces.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.