RLDS Format: Definition, Applications, and Training Data Requirements

RLDS (Reinforcement Learning Datasets) is a data specification, storage format, and toolchain for sequential decision-making datasets, developed by Google DeepMind and built on the TensorFlow Datasets (TFDS) infrastructure. It provides a standardized schema for representing episodes of agent-environment interaction — the fundamental data structure in reinforcement learning, imitation learning, and robot learning — along with tools for dataset creation, validation, transformation, and consumption.

The RLDS schema represents a dataset as a collection of episodes, where each episode is an ordered sequence of steps. Each step contains a fixed set of fields: an observation (the sensory state perceived by the agent), an action (the command executed), a reward (the scalar feedback signal), a discount factor, and boolean flags indicating whether the step is the first or last in the episode. Observations and actions are nested dictionaries that accommodate the heterogeneous sensor suites and control interfaces of different robot platforms — a single observation might contain multiple camera images at different resolutions, a proprioceptive state vector, a language instruction string, and a task embedding vector.

The significance of RLDS for robot learning cannot be overstated. Before its adoption, every research group stored robot data in custom formats with undocumented schemas, making cross-lab data sharing effectively impossible without significant engineering effort. The Open X-Embodiment project (Embodiment Collaboration, 2023) demonstrated what standardization enables: by converting 60+ existing datasets from 22 different robot platforms to RLDS format, the team created a unified training mixture of over 1 million real robot episodes. This mixture trained the RT-X models, which showed that cross-embodiment pretraining on diverse RLDS data improved policy performance on individual robots by 50% on average compared to training on that robot's data alone.

RLDS is implemented on top of TensorFlow Datasets, which provides versioning (each dataset version is immutable and reproducible), documentation (dataset cards with statistics and descriptions), distribution (datasets can be loaded with a single function call from Google Cloud Storage or local storage), and parallel processing (Apache Beam pipelines for efficient dataset generation). The TFRecord storage format supports random access, parallel reads, and efficient compression. While RLDS is natively TensorFlow-compatible, bridge libraries like dlimp and oxe_torch_dataloader provide PyTorch DataLoader interfaces with comparable performance, and conversion to HDF5 or zarr is straightforward for teams that prefer native PyTorch storage.

Converting existing robot data to RLDS format requires understanding both the RLDS schema and your source data's conventions. The RLDS builder API expects data organized as episodes containing steps, where each step has an observation dict, action array, reward scalar, and terminal flags. Common conversion challenges include: identifying episode boundaries in continuous recording logs (typically using task success/failure signals or time gaps), aligning observations and actions that were recorded at different rates (requiring interpolation or nearest-timestamp matching), and mapping robot-specific state representations to the RLDS observation and action schemas.

For new data collection, building RLDS-native recording pipelines avoids conversion overhead. The RLDS EnvLogger wraps a Gym or dm_env environment and records directly to RLDS format. For teleoperation data where a human controls the robot, custom recording scripts can write RLDS steps in real time by capturing observations and actions at each control timestep. Quality validation should run during recording — checking that episode lengths are within expected ranges, observations contain valid data (no NaN values, images are not black), and actions are within the robot's kinematic limits.

The choice of observation and action representations within RLDS affects downstream training. Camera images can be stored at full resolution or resized; the tradeoff is storage size versus information loss. Proprioceptive states should include all available signals (joint positions, velocities, torques, end-effector pose) with documented units and coordinate frames. Actions should use the representation expected by the training pipeline — absolute joint positions, delta joint positions, or end-effector space commands. The Open X-Embodiment project standardized on delta end-effector actions for cross-embodiment training, but robot-specific datasets often use joint-space representations for platform-specific fine-tuning.

Claru delivers datasets in fully validated RLDS format with comprehensive documentation, enabling direct consumption by RT-X, Octo, OpenVLA, and custom training pipelines. Our RLDS builder scripts produce immutable, versioned datasets with dataset cards, per-episode metadata, and observation-action schema documentation. For teams that require PyTorch-native formats, we provide parallel deliveries in HDF5 and zarr with identical data and metadata.

Claru delivers robot learning datasets in fully validated RLDS format, ready for direct consumption by the Open X-Embodiment training infrastructure and generalist robot policies including RT-X, Octo, OpenVLA, and pi-zero. Our RLDS builder pipeline produces immutable, versioned datasets with comprehensive dataset cards documenting robot platforms, sensor specifications, task descriptions, and collection protocols. Each dataset includes per-episode metadata — task success labels, quality scores, operator IDs, and environment configurations — enabling fine-grained data curation and filtering for training. Observation and action schemas follow Open X-Embodiment conventions with documented units, coordinate frames, and robot-specific parameters. Schema validation, temporal alignment verification, and observation completeness checks run automatically before delivery. For teams using PyTorch, we provide parallel deliveries in HDF5 and zarr formats with identical data and metadata. With 3M+ annotated clips across diverse manipulation, navigation, and interaction tasks, Claru provides the RLDS-formatted data at the scale that cross-embodiment foundation models require.