Menu

[Meta FAIR + NYU] Unbundling Multimodal Vision Encoding: training 200+ MLLMs to find encoder pretraining is optional but vision-only parameters are not [R]
📰

[Meta FAIR + NYU] Unbundling Multimodal Vision Encoding: training 200+ MLLMs to find encoder pretraining is optional but vision-only parameters are not [R]

Reading 0:00
15s threshold

X threads (with videos!):

https://x.com/choang333/status/2121335484236386574

https://x.com/_ellisbrown/status/2121335477244428419

Paper: https://www.alphaxiv.org/abs/2611.17334

Blog and FAQ: https://unbundling.github.io

Labs disagree on vision encoders right now. Gemma 4 12B dropped the encoder, Kimi K3 trains its encoder from scratch, and Inkling 2 Small brought an encoder back. Published models can't settle this, because they differ in data and compute and rarely account for encoder pretraining. A standard encoder also bundles three choices: separate pretraining, parameters that only vision tokens use, and placement before the LLM. We tested each choice separately.

Setup:

200+ models, all trained from scratch (LLM included) on the same data.

Lifetime compute: encoder pretraining counts against the same budget as multimodal training.

Main comparisons use Qwen3-1.7B and 4B configurations with 0.4B parameters of vision capacity.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.