How to Build a Cross-Embodiment Robot Dataset

A cross-embodiment dataset is only useful if it contains semantically equivalent tasks performed by different robots. Start by defining a shared task set of 5-15 tasks that all target robots can physically execute, plus optional per-robot task extensions.

For the shared set, choose tasks that test fundamental manipulation capabilities: pick-and-place (single object to target zone), drawer open and close, button pressing, object reorientation, and stacking. For each shared task, write a single task specification that is robot-agnostic: describe the initial conditions (e.g., 'object centered on table within 15cm radius of robot base center'), success criteria ('object placed within 3cm of target marker and stationary for 0.5 seconds'), and failure conditions. Crucially, define the task in terms of end-effector outcomes, not joint-space motions.

For each robot platform, document the kinematic configuration: URDF file path, joint limits, end-effector frame definition, gripper type and range, control frequency, and workspace boundaries. Create a per-robot config file: `robot_config_franka.json` containing `{"dof": 7, "ee_frame": "panda_hand_tcp", "gripper_range_m": [0.0, 0.08], "control_hz": 20, "workspace": {"x": [-0.4, 0.4], "y": [0.2, 0.8], "z": [0.0, 0.5]}}`. This config is used during data collection and again during normalization.

Map the workspace boundaries across robots. The physical workspace may differ (Franka reaches 0.855m, UR5e reaches 0.85m, but their mounting positions may put different table regions in reach). Identify the common reachable workspace and restrict all shared task initial conditions to this overlap zone. This ensures that task difficulty is comparable across robots -- a pick-and-place at the edge of one robot's workspace but the center of another's is not a fair comparison.

Design 2-5 per-robot extension tasks that exploit unique capabilities. For a dual-arm ALOHA: bimanual handoff, collaborative assembly. For a mobile Stretch: navigate-then-manipulate, drawer open from different approach angles. These tasks increase dataset diversity and allow the foundation model to learn robot-specific affordances.

The core engineering challenge of cross-embodiment datasets is mapping different robots' proprietary sensor data and control interfaces to a common representation. The RT-X project established conventions that have become the standard.

For actions, normalize to 7-dimensional end-effector deltas: (dx, dy, dz, droll, dpitch, dyaw, gripper_delta) in the robot's base frame. Compute these from the robot's native action space using forward kinematics. For a Franka controlled in joint space: at each timestep, compute the current and next end-effector poses using `robot.fkine(q_current)` and `robot.fkine(q_next)`, then compute the delta as the relative SE(3) transformation. Express rotation deltas as axis-angle or Euler angles (matching the convention your target model expects -- Octo uses axis-angle, OpenVLA uses Euler). Normalize gripper commands to a 0-1 range: `gripper_normalized = (gripper_width - gripper_min) / (gripper_max - gripper_min)`.

For observations, define a common observation dictionary: `image` (uint8, H x W x 3 -- resize all cameras to a common resolution, typically 256x256 or 224x224), `wrist_image` (uint8, same resolution if available), `state` (float32, robot proprioception), and `language_instruction` (string). The `state` vector should follow the convention: [ee_x, ee_y, ee_z, ee_quat_x, ee_quat_y, ee_quat_z, ee_quat_w, gripper_width]. This 8-dimensional state vector is computable from any robot's joint state via forward kinematics and is the input that cross-embodiment models consume.

For image observations, address camera differences across robots. Wrist cameras on different robots have different intrinsics, mounting positions, and fields of view. Do not simply concatenate images from different cameras -- instead, standardize by cropping to the workspace-relevant region and resizing to 256x256. Apply minimal preprocessing (only resize and center-crop, no color augmentation at dataset-creation time). Store the original camera intrinsics in metadata so downstream users can undo the crop if needed.

Include the robot's embodiment identifier as a per-episode metadata field: `embodiment_id = "franka_panda"`. This enables embodiment-conditional architectures that route through different adapter layers per robot. The Open X-Embodiment convention uses a string identifier; register yours in a `EMBODIMENT_REGISTRY.json` to avoid collision with the official OXE dataset names.

Run parallel collection campaigns across all robot platforms, collecting the shared task set on every robot and the per-robot extensions on their respective platforms. Coordination across platforms is critical to maintaining dataset consistency.

For each robot, set up the collection environment to be as similar as possible. Use the same table, same objects (from the YCB object set or a shared custom object set), and similar lighting conditions. Photograph each setup from a fixed overhead position to document the workspace layout. While exact replication across labs is impossible (different rooms, slightly different table heights), matching the key variables reduces confounding factors.

Use the best available teleoperation interface for each robot. For Franka, GELLO provides the most natural bilateral control. For UR5e, the UMI (Universal Manipulation Interface) gripper-centric approach works well. For ALOHA, use the built-in leader-follower teleoperation. The teleoperation interface will differ across robots, and this is acceptable -- the goal is to produce high-quality demonstrations on each platform, not to standardize the collection method.

Run the shared tasks in a round-robin schedule: collect 50 demonstrations of Task 1 on Robot A, then 50 of Task 1 on Robot B, then 50 of Task 1 on Robot C, before moving to Task 2. This ensures that if collection is interrupted, you have balanced coverage across robots for completed tasks rather than 500 demonstrations on one robot and zero on another.

Record data in each robot's native format (rosbag2 for ROS-based systems, HDF5 for robomimic-based systems). Convert to the common representation in a separate post-processing step rather than recording directly in the common format. This preserves the full-fidelity robot-specific data and lets you fix normalization bugs without re-collecting.

Target 200-500 demonstrations per robot per shared task for fine-tuning existing foundation models. For training a new model from scratch, aim for 1,000+ per robot per task. Include 10-20% failure demonstrations per task per robot -- failure modes differ across embodiments and these differences are training signal, not noise.

Convert all collected data to the RLDS format, the standard consumed by cross-embodiment foundation models (RT-X, Octo, OpenVLA). RLDS stores episodes as TFRecord files where each episode is a sequence of steps, and each step contains observations, actions, reward, and metadata.

Create a dataset builder by subclassing `tfds.core.GeneratorBasedBuilder`. Define the feature specification matching the Open X-Embodiment conventions:

```python features = tfds.features.FeaturesDict({ 'steps': tfds.features.Dataset({ 'observation': tfds.features.FeaturesDict({ 'image': tfds.features.Image(shape=(256, 256, 3)), 'wrist_image': tfds.features.Image(shape=(256, 256, 3)), 'state': tfds.features.Tensor(shape=(8,), dtype=tf.float32), }), 'action': tfds.features.Tensor(shape=(7,), dtype=tf.float32), 'reward': tfds.features.Scalar(dtype=tf.float32), 'is_first': tf.bool, 'is_last': tf.bool, 'is_terminal': tf.bool, 'language_instruction': tfds.features.Text(), }), 'episode_metadata': tfds.features.FeaturesDict({ 'embodiment': tfds.features.Text(), 'task': tfds.features.Text(), 'success': tf.bool, }), }) ```

Write a conversion script per robot platform that reads the native format and yields episodes in the RLDS schema. For each robot, load the raw episode, apply the action normalization from Step 2, resize images to 256x256, compute the 8-dimensional state vector from joint states, and emit one step per timestep. Handle camera availability differences: if a robot lacks a wrist camera, fill `wrist_image` with zeros and document this in the metadata.

Validate the converted dataset by loading it with `tfds.load('your_dataset')` and checking: all tensors have the expected shape, action ranges fall within [-1, 1] after normalization (clip outliers and log warnings for values exceeding [-2, 2]), images are correctly oriented and not corrupted, and language instructions are populated for every step. Run a visualization script that renders one episode per robot per task to visually confirm the data looks correct.

For large datasets (100K+ episodes), use Apache Beam for parallel conversion: `python build_dataset.py --pipeline_type=beam --beam_runner=DirectRunner --num_workers=8`. This parallelizes the IO-heavy conversion across multiple CPU cores and handles fault tolerance (resuming from the last successful episode if the job is interrupted).

After conversion, verify that the dataset is actually useful for cross-embodiment training by checking statistical consistency and running diagnostic training experiments.

Compute per-embodiment statistics and check for distributional alignment. For actions: plot the distribution of each action dimension across embodiments. The x, y, z deltas should have similar ranges if the workspace normalization from Step 2 was correct. If Franka actions have 3x the range of UR5e actions on the same pick-and-place task, the normalization has a scaling error. Use `np.histogram(actions[:, dim], bins=50)` per dimension per embodiment and compare. Rotation action distributions will differ more (different kinematics cause different rotation strategies for the same task) -- this is expected and acceptable.

For observations: compute the mean and variance of the proprioceptive state vector per embodiment. End-effector positions should cluster in the shared workspace region. If one robot's states are offset by 0.5m on the x-axis, the base frame alignment is wrong. Compute t-SNE embeddings of a random sample of images per embodiment. If the embeddings cluster by robot rather than by task, the visual domains are too different for naive cross-embodiment training and you may need visual domain randomization.

Run a diagnostic training experiment. Train a simple behavioral cloning model (2-layer MLP on state + ResNet-18 on images) on three dataset configurations: (A) Franka-only data, (B) UR5e-only data, and (C) pooled cross-embodiment data. Evaluate each model on held-out episodes from both robots. If configuration (C) outperforms (A) and (B) on the other robot's test set, the cross-embodiment data provides positive transfer. If (C) is worse than per-robot training, the action normalization or observation alignment has a problem that must be fixed before scaling.

Check language instruction consistency. For the same task across robots, the language instruction must be identical. If Franka episodes say 'pick up the red block' and UR5e episodes say 'grasp the red cube', the language conditioning will not transfer across embodiments. Normalize all language instructions to a canonical form per task and verify programmatically.

Package the dataset for consumption by foundation model training pipelines. The release should enable any lab to download the data and begin training within an hour.

Structure the release as a TFDS dataset with named configs per embodiment: `your_dataset/franka_panda`, `your_dataset/ur5e`, `your_dataset/aloha`, and `your_dataset/all` (the combined dataset). Each config contains the episodes for that embodiment with consistent schemas. Users can load a single embodiment (`tfds.load('your_dataset/franka_panda')`) or the full pool (`tfds.load('your_dataset/all')`).

Generate comprehensive metadata. For the dataset-level: total episodes per embodiment, total steps per embodiment, task distribution per embodiment, success rates per task per embodiment, and action space statistics (min, max, mean, std per dimension per embodiment). For the per-episode level: embodiment ID, task name, success flag, episode length, and operator ID. For the per-step level: all observation and action tensor shapes and dtypes.

Create a data card following the Open X-Embodiment template. Include: collection setup photos for each robot, URDF or kinematic descriptions, sensor specifications (camera models, proprioceptive sensor specs), action normalization formulas with derivations, language instruction vocabulary, known limitations (e.g., 'UR5e demonstrations have 20% higher grasp failure rate due to gripper compliance'), and intended use cases.

Provide integration code for the major foundation models: a one-file adapter that loads your dataset into Octo's training pipeline, a config file for OpenVLA fine-tuning, and an example training script that runs 100 steps of cross-embodiment BC to verify the pipeline works. Test these integrations yourself before releasing.

Host the dataset on a cloud storage bucket (GCS or S3) with public read access and register it with TFDS so users can download with `tfds.load()`. For very large datasets (> 100GB), also provide a torrent or academic download link. Publish the dataset alongside a technical report (even a 4-page workshop paper) describing the collection protocol, normalization choices, and baseline results. A dataset without published baselines sees 70% less adoption than one with baselines.