Diffusion Transformers (DiTs): Merging generative diffusion processes with Transformer scaling to power next-generation visual AI.
How replacing convolutional U-Nets with Transformer backbones revolutionized visual synthesis, video generation, and spatial AI.
For years, generative image and video models relied almost entirely on the U-Net architecture. In early diffusion models, U-Nets handled the complex task of denoising image latent vectors step-by-step through downsampling and upsampling convolutional layers.
While U-Nets were effective for moderate resolutions, they suffered from a major scaling limitation: convolutional layers struggle to capture global context across high-resolution spatial dimensions and long video sequences.
To unlock true compute scaling for visual media, researchers engineered Diffusion Transformers (DiTs)—replacing convolutional U-Nets with Transformer backbones.
What is a Diffusion Transformer?
A Diffusion Transformer (DiT) combines two powerful ideas in modern…