Diffusion Transformer (DiT): Scalable Denoising for Video Generation and Robot Control

The Diffusion Transformer (DiT) is a neural network architecture that replaces the U-Net backbone traditionally used in denoising diffusion models with a Transformer backbone. Introduced by Peebles and Xie (2023), DiT processes inputs by patchifying them into token sequences and applying a stack of Transformer blocks with self-attention and feed-forward layers — the same architecture used in Vision Transformers (ViT) and GPT. Conditioning on the diffusion timestep and class labels is achieved through adaptive layer normalization (adaLN-Zero), where the normalization parameters are predicted from the conditioning signal.

The motivation for DiT is scaling. U-Nets have an irregular architecture with asymmetric encoder/decoder paths, skip connections at multiple resolutions, and a mix of convolutions and attention at different scales. This complexity makes it difficult to predict how performance will change as the model grows. Transformers, by contrast, have uniform repeated blocks with well-characterized scaling behavior — loss decreases as a smooth power law of model size and training data (Kaplan et al., 2020). DiT inherits this predictability: Peebles and Xie showed that FID on ImageNet improved monotonically from 68.4 (DiT-S, 33M params) to 2.27 (DiT-XL/2, 675M params) with no irregular jumps.

For video generation, DiT scales to the temporal dimension by treating video frames as additional spatial patches in the token sequence. A video of T frames at resolution HxW with patch size P produces T*H*W/P^2 tokens, processed by the same Transformer blocks. This unified spatial-temporal attention, or factored variants (spatial attention + temporal attention in alternating layers), powers commercial video generation models. OpenAI's Sora, Stability AI's Stable Video Diffusion, and Runway Gen-3 all use DiT-class architectures for video generation.

In robotics, DiT serves as the backbone for scalable action generation. Physical Intelligence's pi-zero uses a flow-matching variant of DiT where the action denoising process is modeled as an ordinary differential equation (ODE) rather than a stochastic process, enabling fewer function evaluations (5-10 vs. 100+ for DDPM) and faster inference. The Robotics Diffusion Transformer (RDT, Liu et al., 2024) demonstrated that a 1.2B-parameter DiT trained on multi-robot data from Open X-Embodiment produces broadly capable manipulation policies, with performance improving smoothly as model and data scale increase — the first demonstration of Transformer-like scaling laws for robot control.

For teams building robot learning systems, DiT's practical implications are twofold: it enables scaling to larger models and datasets, but it also increases computational requirements for both training and inference.

On the training side, DiT-based robot policies require GPU infrastructure comparable to medium-scale language model training. A 675M-parameter DiT policy training on 1M demonstration episodes for 500K steps requires approximately 256 A100 GPU-hours. This is 5-10x more compute than a U-Net-based Diffusion Policy of comparable capability on single-task benchmarks. The payoff comes from multi-task and cross-embodiment generalization: the same DiT model can handle dozens of tasks across multiple robots, amortizing the training cost.

Data requirements for DiT are higher in total volume but lower per task. A U-Net Diffusion Policy might need 200 demonstrations for a single task. A DiT policy trained on 50 tasks needs 5,000-10,000 total demonstrations (100-200 per task), but the per-task data requirement is similar because shared visual and motor representations transfer across tasks. The data should be formatted as (observation, action) trajectories with consistent action representations (7D Cartesian delta is standard) and task language annotations for conditioning.

Inference optimization is critical for deployment. A 675M DiT with 100 DDPM steps is far too slow for real-time control (~0.5 Hz). Three optimization paths exist: (1) DDIM sampling with 10-16 steps (5-10 Hz); (2) flow matching with 5-10 ODE steps, as used by pi-zero (5-15 Hz); (3) consistency distillation to 1-2 steps (15-30 Hz). For tabletop manipulation at 10 Hz control, flow matching is the current best tradeoff between quality and speed. For higher-frequency tasks (20+ Hz), consistency distillation or smaller DiT variants are necessary.

The data pipeline for DiT training should emphasize diversity: diverse tasks, diverse environments, diverse object categories, and (for cross-embodiment models) diverse robot platforms. Unlike single-task models where demonstration quality is paramount, DiT's large capacity allows it to benefit from noisier data mixed with clean data — the model learns to extract the signal while being robust to quality variation. However, a quality floor still applies: demonstrations with task failures, dropped objects, or operator confusion should be filtered out.

Claru provides the diverse, multi-task training data that DiT-based robot policies need to realize their scaling potential. Our demonstration datasets span manipulation tasks across multiple robot platforms (Franka, UR5, xArm, ALOHA) with standardized 7D Cartesian delta actions and natural language task annotations — the exact data format consumed by RDT, pi-zero, and other DiT-based architectures.

The key value Claru adds to DiT training is data diversity at scale. DiT models improve most when training data covers diverse environments, objects, and manipulation strategies. Our collection network of 10,000+ operators in 100+ cities provides the environmental breadth that lab-collected datasets lack. Combined with public cross-embodiment datasets like OXE, Claru's data fills the diversity gap that limits generalization in current DiT robot policies. Datasets ship in RLDS format with quality scores, enabling teams to set their own quality-diversity tradeoff during training data curation.