LeRobot Format: Complete Guide for Robotics Data

LeRobot datasets are stored on Hugging Face Hub with a specific structure: episodes stored in Parquet files with observation columns (images as paths, proprioceptive state as arrays) and action columns. Video frames are stored separately as MP4 files referenced by episode/frame indices. Metadata is in a dataset_info.json with robot config, fps, and feature descriptions.

The LeRobot v2 dataset format represents a significant evolution in how robotics data is organized. Each dataset consists of three core components: a metadata directory containing info.json (dataset-level configuration including fps, robot type, and feature definitions), episode Parquet files partitioned by episode index, and video directories organized by camera name. The Parquet files store one row per timestep with columns for episode_index, frame_index, timestamp, state vectors (as fixed-length arrays), action vectors, and boolean flags like next.done. Image observations are not embedded in the Parquet files but instead stored as MP4 video files at videos/{camera_name}/episode_{index}.mp4, with the Parquet rows providing frame-level indexing into these videos.

This separation of tabular data (Parquet) from visual data (MP4) is a deliberate design choice that reduces storage by 90%+ compared to per-frame image storage, while enabling efficient video compression across temporal frames. The info.json file defines every feature's dtype, shape, and names array (e.g., joint names for state vectors), making datasets fully self-describing. LeRobot v2 also introduced a standardized task column and episode-level task mapping, enabling multi-task datasets where different episodes correspond to different instructions.

Loading a LeRobot dataset from Hugging Face Hub is a single function call: from lerobot.common.datasets.lerobot_dataset import LeRobotDataset, then dataset = LeRobotDataset('lerobot/aloha_sim_transfer_cube_human'). This returns a PyTorch-compatible dataset object where dataset[i] yields a dictionary with state, action, and decoded image tensors for timestep i. The framework handles video decoding, Parquet reading, and timestamp alignment transparently. For streaming access without full download, pass streaming=True to load data on-the-fly from Hub storage.

Creating a new LeRobot dataset involves the LeRobotDataset.create() factory method. You specify the robot type, fps, and feature definitions, then call dataset.add_frame() for each timestep with a dictionary of observations and actions. At episode boundaries, call dataset.save_episode() to flush the current episode's Parquet data and compress video frames. The framework automatically generates info.json, handles MP4 encoding with configurable quality (crf parameter, default 20), and structures the Hub-compatible directory layout. After recording, push to Hub with dataset.push_to_hub('your-org/your-dataset'). The lerobot CLI also provides lerobot/scripts/push_dataset_to_hub.py for batch conversion from existing formats.

Migrating to LeRobot format involves three steps: defining the feature schema, converting episode data, and publishing to Hub. Start by creating an info.json that maps your observation and action spaces to LeRobot's feature format. Each feature needs a dtype (float32, int64, or video for images), shape (e.g., [7] for a 7-DoF action space), and names array listing each dimension. The robot_type field should match one of LeRobot's supported configurations (aloha, koch, so100, etc.) or use a custom string for unsupported hardware.

For the data conversion itself, the most common pattern is reading from your source format episode by episode, calling LeRobotDataset.add_frame() with the observation and action data for each timestep, and calling save_episode() at episode boundaries. Image observations should be passed as numpy uint8 arrays (H, W, 3); the framework handles MP4 encoding internally. The critical detail is matching your source fps to the target fps specified in info.json. If your source data was collected at 30 Hz but your target policy expects 10 Hz, downsample during conversion rather than at training time.

After conversion, validate the dataset by loading it back with LeRobotDataset and checking that episode counts, frame counts, and feature shapes match your expectations. Run the built-in visualization with python -m lerobot.scripts.visualize_dataset --repo-id your-org/your-dataset to render episodes as video. Push to a private Hub repo for team access, or make it public to contribute to the growing LeRobot dataset ecosystem, which already includes over 200 community-contributed datasets across dozens of robot embodiments.

Claru delivers datasets in LeRobot format ready to push to Hugging Face Hub or load locally. We configure the info.json for your specific embodiment, with proper camera indices, action space definitions, and task descriptions.

Every delivery includes validated Parquet files with correct episode indexing, MP4 videos encoded at configurable quality levels, and a complete info.json with your robot's joint names and feature specifications. We test each dataset by loading it with LeRobotDataset and running a training smoke test with ACT and Diffusion Policy to verify compatibility. For teams with private Hub organizations, Claru can push directly to your repository with appropriate access tokens. We also provide a conversion script for teams that need the same data in RLDS or HDF5 format alongside the LeRobot delivery.