X threads (with videos!):
https://x.com/choang333/status/2121335484236386574
https://x.com/_ellisbrown/status/2121335477244428419
Paper: https://www.alphaxiv.org/abs/2611.17334
Blog and FAQ: https://unbundling.github.io
Labs disagree on vision encoders right now. Gemma 4 12B dropped the encoder, Kimi K3 trains its encoder from scratch, and Inkling 2 Small brought an encoder back. Published models can't settle this, because they differ in data and compute and rarely account for encoder pretraining. A standard encoder also bundles three choices: separate pretraining, parameters that only vision tokens use, and placement before the LLM. We tested each choice separately.
Setup:
200+ models, all trained from scratch (LLM included) on the same data.
Lifetime compute: encoder pretraining counts against the same budget as multimodal training.
Main comparisons use Qwen3-1.7B and 4B configurations with 0.4B parameters of vision capacity.…