Training Data for Pi-Zero (pi0)

Claru provides pi0-compatible data with synchronized multi-view RGB at 60 Hz (exceeding pi0's 50 Hz minimum), full proprioceptive recordings (joint positions, velocities, torques, end-effector wrenches), and hierarchical language annotations at task, step, and timestep granularity. Our collection infrastructure uses hardware-triggered frame synchronization to ensure sub-millisecond alignment across all sensor modalities — a strict requirement for pi0's action chunk training. We deliver in RLDS and LeRobot HDF5 formats with per-embodiment action normalization, camera calibration metadata, and episode-level quality scores. For teams bootstrapping cross-embodiment pre-training, Claru can supply diverse demonstration data across multiple robot platforms to seed the pre-training phase before task-specific fine-tuning.

Pi-Zero (pi0) is a vision-language-action (VLA) model introduced by Physical Intelligence in October 2024. It is the first generalist robot policy to demonstrate competence across a wide range of dexterous manipulation tasks — from folding laundry and assembling boxes to bussing tables — using a single set of weights. The key insight behind pi0 is that a pre-trained vision-language model (VLM) can serve as a universal robot backbone when coupled with an expressive action generation mechanism.

Unlike prior VLAs such as RT-2 or OpenVLA that tokenize actions into discrete bins, pi0 generates continuous action trajectories via flow matching. Flow matching learns a vector field that maps a Gaussian noise distribution to the target action distribution in a single forward pass at inference time. This avoids the lossy discretization of high-dimensional action spaces and is significantly faster than iterative diffusion sampling, enabling real-time 50 Hz control on physical hardware.

The pi0 architecture begins with a 3-billion-parameter PaLI-based VLM backbone pre-trained on internet-scale image-text data. During robot pre-training, the model ingests multi-view RGB images and a natural language instruction, encodes them through the VLM trunk, and passes the resulting latent representation to a flow matching action head. The action head outputs action chunks — sequences of 50 future waypoints at 50 Hz — that are sent directly to the robot's low-level controller.

Physical Intelligence demonstrated pi0 on seven distinct robot embodiments including bi-manual ALOHA setups, single-arm Franka Pandas, and mobile manipulators. The paper reports that pi0 achieves state-of-the-art performance on standard benchmarks and, more importantly, solves real-world dexterous tasks that no prior generalist policy had addressed, such as folding multiple garment types and loading a dishwasher from a cluttered counter.

The core innovation in pi0 is the marriage of a pre-trained VLM with a flow matching action head. The VLM backbone — a PaLI variant with roughly 3B parameters — processes interleaved image and text tokens. Rather than auto-regressively decoding discretized action tokens (as in RT-2), pi0 attaches a lightweight MLP action head that is trained with the flow matching objective from Lipman et al. (2023). During training, the head learns to denoise a randomly sampled noise vector into the target action chunk, conditioned on the VLM's latent representation. At inference, a single forward pass through the denoising network produces the full 50-step action trajectory.

This design has three practical consequences for data. First, because action generation is continuous, there is no discretization bottleneck — the model can represent the full precision of the action signal, which matters for tasks like cloth folding where sub-millimeter gripper precision is needed. Second, the flow matching head is much cheaper to train than a full diffusion model; a single denoising step replaces the 50-100 steps typical in DDPM-based policies. Third, the action chunk formulation naturally handles temporal coherence — the 50-step output is a smooth trajectory, not a sequence of independent decisions.

Cross-embodiment training is achieved through embodiment-specific action heads that share the same VLM backbone. Each robot platform defines its own action dimensionality and normalization scheme, but the visual-language understanding is shared. This means that demonstrations from a Franka Panda performing pick-and-place tasks improve the model's language grounding even for an ALOHA robot performing different tasks. The shared trunk acts as a universal visual-semantic encoder, while the per-embodiment heads translate that understanding into hardware-specific motor commands.

Physical Intelligence also introduced a post-training fine-tuning recipe where pi0 is further tuned on a narrow task distribution with as few as 100-200 demonstrations. This fine-tuning stage uses a lower learning rate and shorter training schedule, and the paper shows that it consistently improves performance on the target task without degrading the model's generalist capabilities.

Pi-zero's training proceeds in three phases, each with distinct data requirements. Phase 1 is VLM pre-training on internet-scale image-text data (billions of image-text pairs). This phase uses the standard PaLI pre-training mix and is not specific to robotics. Phase 2 is cross-embodiment robot pre-training, where the model is trained on a diverse mixture of robot demonstration datasets spanning multiple hardware platforms. Physical Intelligence used a proprietary dataset of over 10,000 demonstrations across 7+ embodiments, supplemented by publicly available datasets. Phase 3 is task-specific fine-tuning on 100-1,000 high-quality demonstrations per target task.

For teams building their own pi0-style pipelines, the critical data requirements center on Phase 2 and Phase 3. Robot demonstrations must include synchronized multi-view RGB images (typically 2-3 camera angles), continuous joint-state trajectories recorded at 50 Hz or higher, natural language task descriptions, and optionally proprioceptive signals such as joint torques and end-effector force/torque readings. The multi-view requirement is essential because pi0 uses cross-view attention to build spatial representations — a single wrist camera is insufficient for most manipulation tasks.

Action labels must be continuous and high-frequency. Unlike RT-1 or RT-2 which operate at 3 Hz with discretized actions, pi0 expects dense 50 Hz joint position or end-effector delta trajectories. This means teleoperation systems must record at at least 50 Hz with sub-degree joint accuracy. For bi-manual setups (e.g., ALOHA), each demonstration contains a 24-dimensional action vector per timestep (7 joints + gripper per arm, times two arms), resulting in approximately 1,200 action dimensions per second of demonstration.

Data diversity is more important than per-task volume. The paper demonstrates that cross-embodiment pre-training with heterogeneous data from multiple robot platforms significantly improves generalization, even when the target robot is not represented in the pre-training mix. For fine-tuning, the paper reports strong results with as few as 150 demonstrations per task when building on the pre-trained checkpoint, but emphasizes that demonstration quality — measured by task completion rate, smooth trajectories, and absence of pauses or recoveries — is critical.

Claru provides data compatible with pi0's demanding multi-view, high-frequency requirements. Our collection infrastructure captures synchronized multi-camera RGB streams at 60 Hz with hardware-triggered frame alignment, exceeding pi0's 50 Hz minimum. We record full proprioceptive state — joint positions, velocities, torques, and end-effector wrenches — alongside the visual streams, enabling teams to train with or without auxiliary proprioceptive inputs.

For language conditioning, Claru's annotation pipeline generates hierarchical language descriptions: a concise task instruction (e.g., 'fold the blue shirt'), a step-level narration (e.g., 'grasp the right sleeve, fold it toward center'), and detailed per-timestep commentary where required. This multi-granularity annotation supports both the coarse task-level conditioning pi0 uses during pre-training and the fine-grained language grounding that improves generalization.

Claru delivers datasets in formats directly consumable by pi0-style training pipelines, including RLDS (for TensorFlow-based workflows) and LeRobot HDF5 (for PyTorch). Our conversion pipeline handles action normalization per-embodiment, camera intrinsic/extrinsic calibration metadata, and episode-level quality scores. We also provide data provenance documentation covering collection environment, operator skill level, and hardware configuration — metadata that is essential for diagnosing training failures and understanding distribution shift.