Training Data for Pi-0.5

Claru provides multi-embodiment demonstration data for pi-0.5-compatible pipelines via the openpi framework. Our datasets include hardware-synchronized multi-view RGB at 50 Hz (primary + wrist cameras), proprioceptive state vectors, smooth continuous action trajectories that produce clean FAST tokens, and human-written language instructions. For long-horizon tasks, we annotate reasoning traces with sub-goal descriptions. We deliver pre-computed FAST codebook statistics for the target action space, enabling immediate fine-tuning. Our collection spans single-arm, bi-manual, and mobile manipulation platforms across household, industrial, and dexterous task domains.

Pi-0.5 is a vision-language-action (VLA) model released by Physical Intelligence (pi) in February 2025. It is the successor to pi-zero (pi0), the flow-matching-based generalist robot policy that demonstrated dexterous manipulation on seven embodiments in October 2024. Pi-0.5 represents a significant architectural and data-scale leap: it replaces pi0's flow matching action head with FAST (Fourier-domain Action Sequence Tokenization), moves to a larger VLM backbone, and is trained on what Physical Intelligence describes as the largest and most diverse proprietary robot dataset ever assembled for a single model.

The core innovation in pi-0.5 is the FAST tokenizer, which converts continuous robot action trajectories into discrete token sequences that the VLM backbone can predict using standard autoregressive next-token prediction -- the same mechanism the model uses for language. FAST works by applying a discrete cosine transform (DCT) to action trajectories, quantizing the frequency-domain coefficients into a discrete codebook, and representing each action chunk as a short sequence of tokens. This eliminates the need for a separate action head (flow matching, diffusion, or regression) and allows the entire model to be trained with a single cross-entropy loss, dramatically simplifying the training pipeline.

Pi-0.5 was trained on robot data from more than 10 different robot embodiments including single-arm manipulators (Franka Panda, UR5), bi-manual systems (ALOHA-style dual-arm setups), mobile manipulators, and humanoid robots. The training data includes both teleoperated demonstrations and autonomously collected data. Physical Intelligence has not disclosed exact dataset sizes, but public statements indicate the data is orders of magnitude larger than the Open X-Embodiment dataset (which contains approximately 1 million episodes). The model was evaluated on tasks spanning household manipulation (laundry folding, dishwasher loading), industrial assembly, dexterous manipulation (peg insertion, cable routing), and long-horizon multi-step tasks requiring extended planning.

Pi-0.5's position in the field is as the most capable generalist robot policy demonstrated to date. While pi0 showed breakthrough results on dexterous tasks that no prior model could solve (folding different garment types, box assembly), pi-0.5 extends this to longer-horizon tasks, better generalization across environments, and faster inference. The model is partially open: Physical Intelligence released the 'openpi' inference framework for running pi-0.5-compatible models, but the full model weights and training data remain proprietary.

Pi-0.5's architecture is a vision-language-action model built on a pre-trained VLM backbone (PaLI-based, approximately 3 billion parameters). Multi-view RGB images are encoded by a ViT visual encoder into patch-level tokens. A natural language instruction is tokenized into text tokens. Proprioceptive state (joint angles, gripper state) is projected into the token embedding space. All tokens -- visual, language, and proprioceptive -- are concatenated into a single sequence and processed by the VLM transformer. The model then autoregressively generates FAST action tokens that decode back to continuous 50-step action trajectories.

FAST (Fourier-domain Action Sequence Tokenization) is the key innovation that distinguishes pi-0.5 from both its predecessor pi0 and from other VLAs. Prior VLAs used one of three approaches to generate actions: (1) discretize each action dimension into bins and predict them as tokens (RT-2, OpenVLA) -- lossy and produces quantization artifacts; (2) use a diffusion or flow matching head on top of the VLM -- requires a separate loss and training infrastructure; or (3) regress continuous values directly -- uni-modal and cannot represent multi-modal action distributions. FAST solves all three problems: it converts a chunk of 50 continuous action steps into a short sequence of approximately 8-16 discrete tokens via DCT compression and codebook quantization, which the VLM predicts using the same cross-entropy loss used for language.

The FAST tokenization process works as follows: (1) take a 50-step action trajectory (50 x D, where D is the action dimension); (2) apply a discrete cosine transform (DCT) along the temporal axis, which compresses the trajectory into a few dominant frequency coefficients; (3) quantize the DCT coefficients into entries from a learned discrete codebook; (4) flatten the quantized codes into a token sequence. At inference, the model generates FAST tokens autoregressively, the tokens are de-quantized to DCT coefficients, and the inverse DCT produces a continuous 50-step action trajectory. This gives pi-0.5 the best of both worlds: the training simplicity of token prediction and the expressiveness of continuous action generation.

A second key innovation is pi-0.5's extended reasoning capability. Unlike pi0, which processes a single instruction and immediately generates actions, pi-0.5 can perform chain-of-thought reasoning before acting. For long-horizon tasks ('clean up the kitchen'), the model can break the task into sub-goals, reason about the current scene state, and generate an action plan before executing each step. This reasoning happens in the same token space as language and actions -- the model generates a text plan followed by action tokens, all in one autoregressive pass. This capability was enabled by training on demonstration data that includes annotated reasoning traces alongside actions.

Pi-0.5's training data has two components: web-scale vision-language data for VLM pre-training and multi-embodiment robot data for action fine-tuning. The VLM pre-training uses internet-scale image-text pairs, visual question answering data, and other multimodal datasets -- the same pre-training pipeline used by PaLI-class models. This gives the VLM backbone strong visual understanding, language fluency, and reasoning ability before it ever sees robot data.

The robot training data is the most distinctive aspect of pi-0.5. Physical Intelligence collected what they describe as the largest and most diverse robot manipulation dataset ever assembled for a single model. The data spans 10+ robot embodiments including Franka Panda, UR5, ALOHA bi-manual systems, mobile manipulators with navigation, and humanoid upper-body platforms. Each demonstration is a trajectory of: multi-view RGB frames (up to 3 camera views synchronized at 50 Hz), proprioceptive state (joint positions, velocities, gripper state), a natural language task description, and continuous action labels (end-effector deltas or joint velocity commands, depending on the embodiment).

For teams fine-tuning pi-0.5-compatible models via the openpi framework, the practical data requirements mirror those of other cross-embodiment VLAs but with emphasis on action trajectory quality. Because FAST tokenizes 50-step action chunks, the temporal smoothness and consistency of action trajectories is critical -- jerky teleoperation data, pauses, or discontinuities produce poor FAST tokens. Actions should be recorded at 50 Hz with smooth, continuous trajectories. Multi-view camera feeds should be hardware-synchronized. Language instructions should describe the full task, and for long-horizon tasks, annotated sub-goal descriptions are needed to train the reasoning capability.

Data volume recommendations for fine-tuning: 200-500 demonstrations per task for single-task adaptation on a known embodiment, 1,000-5,000 demonstrations for multi-task training across a new embodiment, and 10,000+ demonstrations for training a new generalist policy across multiple embodiments. The FAST tokenizer needs to be trained (or at least calibrated) on the target action distribution, so the dataset should cover the full range of actions the robot will need to execute.

Claru provides multi-embodiment demonstration data compatible with pi-0.5's input requirements and the openpi fine-tuning framework. Our datasets include multi-view RGB observations (up to 3 synchronized camera feeds at 50 Hz), proprioceptive state vectors, natural language task descriptions, and continuous action trajectories with the temporal smoothness that FAST tokenization requires. We capture all data at 50 Hz native frequency -- matching pi-0.5's action chunk rate -- so no temporal interpolation is needed.

For long-horizon tasks that leverage pi-0.5's reasoning capability, Claru provides annotated reasoning traces alongside action data. Human annotators decompose multi-step tasks into sub-goals and describe the reasoning at each decision point. This annotation format matches the chain-of-thought training data that enables pi-0.5's extended planning. Our annotation protocol covers the task domains Physical Intelligence has demonstrated: household manipulation (laundry, kitchen tasks, tidying), industrial assembly (insertion, cable routing, part mating), and dexterous manipulation (deformable objects, tool use, bimanual coordination).

Claru's collection network spans single-arm (Franka, UR5, xArm), bi-manual (ALOHA-type dual-arm setups), and mobile manipulation platforms. We deliver data in formats compatible with the openpi inference and fine-tuning framework, including pre-computed FAST codebook statistics for the target action space. Our quality pipeline validates temporal smoothness (rejecting trajectories with action discontinuities exceeding configurable thresholds), camera synchronization (sub-20ms cross-view alignment), and instruction quality (human review of all language annotations).