Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data needed to adapt these models may be distributed across institutions or organizations that cannot centralize their raw records.

Federated learning provides a way to coordinate training across these data-local sites. For VLMs, the challenge is not only orchestration. Sites may contribute different task or modality mixes, and model updates can be large enough to strain network bandwidth and server memory.

This post focuses on two design decisions for federated multimodal AI workflows: what model state should cross the network, and how should it be transferred and aggregated efficiently?…