Training Data for CrossFormer

Claru provides cross-embodiment demonstration data from multiple robot platforms in RLDS format with OXE-standard metadata. Our collection spans WidowX 250, Franka Emika, UR5e, and ALOHA configurations, covering the most common embodiments in CrossFormer's pretraining distribution. For fine-tuning on supported embodiments, we deliver 100-500 demonstrations per task with matched camera configurations, image resolutions, and control frequencies. For new embodiment onboarding, we collect the diverse multi-task corpus (200-1,000 episodes across 5+ tasks) needed to train effective tokenizer/detokenizer layers. All data includes per-embodiment action normalization statistics and validated RLDS metadata specifying embodiment name, action dimension, and camera configuration.

CrossFormer is a cross-embodiment robot policy introduced by Doshi, Walke, Mees, Dasari, and Levine at UC Berkeley and Carnegie Mellon University in 2024. It addresses a fundamental challenge in generalist robot learning: different robots have different numbers of cameras, different action spaces (some use 6-DoF end-effector control, others use 7-DoF joint positions, others use 2-DoF base velocity), and different observation modalities. Prior approaches either forced all robots into a lowest-common-denominator representation (losing information) or trained separate models per embodiment (losing cross-embodiment transfer).

CrossFormer's key innovation is a set of embodiment-specific tokenizer and detokenizer layers that sit between the robot's raw observation/action space and a shared transformer backbone. The tokenizer maps each robot's unique observations (variable numbers of camera views, proprioceptive dimensions, and language inputs) into a fixed set of tokens that the shared transformer can process. The detokenizer maps the transformer's output back into the robot's specific action space. The shared transformer learns manipulation knowledge that transfers across embodiments, while the tokenizer/detokenizer layers handle the embodiment-specific translation.

CrossFormer was pretrained on a large subset of the Open X-Embodiment (OXE) dataset — approximately 900K episodes spanning over 20 robot platforms including WidowX 250, Franka Emika, Google Robot, ALOHA, xArm, UR5, and others. After pretraining, CrossFormer can be fine-tuned to a new embodiment using only 50-200 demonstrations, and the fine-tuned policy outperforms policies trained from scratch on the same target data.

The model is open-source and builds on the Octo codebase, making it straightforward to integrate into existing OXE-compatible training pipelines. CrossFormer represents the current state of the art for cross-embodiment transfer in manipulation, outperforming both Octo and RT-2-X on several benchmark tasks when fine-tuned with limited target-embodiment data.

CrossFormer's architecture has three distinct components: embodiment-specific input tokenizers, a shared transformer backbone, and embodiment-specific action detokenizers. During both pretraining and fine-tuning, only the tokenizers and detokenizers for the active embodiment(s) are updated alongside the shared backbone. When fine-tuning on a new embodiment, new tokenizer and detokenizer layers are initialized while the shared backbone is loaded from the pretrained checkpoint.

The input tokenizer for each embodiment consists of a ViT-S/16 image encoder (one per camera view), a linear projection for proprioceptive state, and the shared T5-base language encoder. Each tokenizer maps its inputs into a fixed number of tokens (typically 16-64 tokens per observation) with the same embedding dimension as the shared transformer (768-dim). This fixed-size token representation is what enables heterogeneous inputs to be processed by the same transformer.

The shared transformer backbone is a standard decoder-only transformer with 12 layers, 12 attention heads, and 768-dim hidden states. It processes sequences of observation tokens with causal attention and outputs a set of action tokens. The action tokens are passed to the embodiment-specific detokenizer, which is a small MLP (2 layers, 256 hidden units) that maps the transformer's output to the target action dimension and applies per-embodiment action scaling.

A critical design choice is the action chunk prediction with a diffusion head, inherited from Octo. Rather than predicting a single action, CrossFormer predicts 4-step action chunks using DDPM denoising with 10 diffusion steps. This provides temporal consistency and handles multi-modal action distributions. The diffusion head operates in the detokenizer space, so it is embodiment-specific.

CrossFormer's pretraining uses a dataset mixture strategy that upweights high-quality datasets. BridgeData V2 and DROID receive higher sampling weights than smaller or lower-quality datasets. The mixture is balanced so that each embodiment contributes roughly proportionally to the gradient signal, preventing the largest datasets from dominating. This balancing is critical — without it, the shared backbone overfits to the distribution of the largest embodiment and transfers poorly to smaller ones.

CrossFormer's data requirements depend on whether you are contributing to pretraining, fine-tuning an existing CrossFormer checkpoint, or training tokenizer/detokenizer layers for a new embodiment. Each scenario has different volume and format requirements.

For pretraining contributions, data must be in RLDS format following OXE conventions. Each episode needs: RGB images from one or more cameras (resized to 256x256 before augmentation crops to 224x224), proprioceptive state as a float vector, continuous actions in the robot's native control space, and optionally a natural language instruction. The RLDS metadata must specify the embodiment name, action dimension, proprioceptive dimension, and number of camera views. The OXE dataset builder template provides the required schema.

For fine-tuning CrossFormer to a new task on an already-supported embodiment (e.g., WidowX 250, Franka Emika), you need 50-200 demonstrations of the target task in RLDS format. The Doshi et al. paper shows that 100 demonstrations on a WidowX 250 achieve within 5% of the performance of 500 demonstrations when starting from the CrossFormer pretrained checkpoint. This is 2-3x more sample-efficient than Octo fine-tuning on the same tasks.

For adding a new embodiment not seen during pretraining, you need to train new tokenizer and detokenizer layers. This requires 200-1,000 demonstrations on the new embodiment, covering a variety of tasks (not just the target task). The more diverse the training tasks, the better the tokenizer/detokenizer layers learn to represent the new embodiment's observation and action space. Doshi et al. recommend at least 5 distinct tasks for tokenizer training.

Data quality considerations mirror those of Octo and BridgeData V2: demonstrations should be successful completions of the intended task, with consistent camera viewpoints within an embodiment, and actions in continuous control space (not discretized). Action normalization to zero-mean, unit-variance per dimension is handled by the training pipeline but requires accurate dataset statistics. Language instructions, if present, should use simple imperative sentences that match the style of the OXE pretraining data.

Claru provides cross-embodiment demonstration data from multiple robot platforms, delivered in RLDS format with the OXE-standard metadata that CrossFormer requires. Our current collection capabilities span WidowX 250, Franka Emika, Universal Robots UR5e, and ALOHA bimanual configurations — covering the most common embodiments in the CrossFormer pretraining distribution.

For teams fine-tuning CrossFormer on a supported embodiment, we deliver targeted task-specific datasets of 100-500 demonstrations per task. Each dataset includes the correct RLDS feature specification, per-embodiment action normalization statistics, and language annotations. We match the camera configurations, image resolutions, and control frequencies of the pretraining data for the target embodiment to minimize distributional shift.

For teams onboarding a new embodiment to CrossFormer, we collect the diverse multi-task demonstration corpus (200-1,000 episodes across 5+ tasks) needed to train effective tokenizer/detokenizer layers. We work with your team to define the task distribution and ensure sufficient coverage of the embodiment's workspace and manipulation capabilities.

Our quality pipeline validates cross-embodiment data compatibility: action dimensions match the declared embodiment specification, image resolutions are consistent within each embodiment, proprioceptive state vectors have the correct dimension and value ranges, and language annotations follow OXE conventions. This validation prevents silent data corruption that could degrade the shared transformer backbone during co-training.