Unified multimodal models stop fighting themselves when the generation objective targets the same continuous representation that understanding already uses and UMM-JEPA does this through masked, coarse-to-fine prediction of those features.
The Real Problem / Why Naive Approaches Fail
The pitch for a unified multimodal model is simple. One backbone reads images, writes images and answers questions about them. You expect the two skills to help each other: a model that can draw a dog should understand dogs better.
That isn't what usually happens. Generative training often adds little or nothing to understanding. Teams that bolt an image-generation objective onto a vision-language model frequently find their VQA and captioning numbers flat or slightly worse, even though the generation samples look good.
The UMM-JEPA abstract gives a plausible diagnosis. The two tasks optimize different representation spaces.β¦