RLDS (Reinforcement Learning Datasets): Complete Guide for Robotics Data

RLDS stores trajectories as sequences of (observation, action, reward, discount, is_terminal) tuples in TensorFlow's tf.data.Dataset format, serialized as TFRecord files. Each episode is a separate entry with nested feature dictionaries for images, proprioceptive state, and action vectors. Metadata is stored in dataset_info.json alongside the TFRecord shards.

The core data model revolves around two nested levels: episodes and steps. Each episode is a tf.data.Dataset of steps, and the full dataset is a tf.data.Dataset of episodes. A step contains an observation dictionary (which can hold images as uint8 tensors, joint positions as float32 arrays, gripper state, and natural language instructions), an action array, a scalar reward, a discount factor, and boolean flags for is_first, is_last, and is_terminal. This two-level nesting distinguishes RLDS from flat dataset formats and makes it natural to express the sequential structure of robot demonstrations.

Feature connectors in the dataset_info.json file define the dtype, shape, and encoding for every field. Images can be stored as raw tensors or JPEG/PNG-compressed bytes with a tfds.features.Image connector that handles decoding transparently. The dataset builder pattern means that adding a new dataset to the RLDS ecosystem requires implementing a single Python class with a _generate_examples method, and the TFDS infrastructure handles sharding, checksumming, and version management automatically.

Loading an RLDS dataset requires just two lines: call tfds.load('your_dataset') to get a tf.data.Dataset of episodes, then iterate over episodes and their steps. For example, `dataset = tfds.load('bridge_dataset', split='train')` returns the full BridgeData V2 collection in RLDS format. Each episode is accessed via `for episode in dataset: for step in episode['steps']:`, where step['observation']['image'] yields a decoded image tensor and step['action'] yields the 7-DoF action vector.

Writing a new RLDS dataset involves creating a TFDS DatasetBuilder subclass. You define a features dictionary specifying every field's type and shape, implement _split_generators to point to raw data, and implement _generate_examples to yield (episode_key, episode_data) pairs. The envlogger library from Google DeepMind can also record RLDS datasets directly from a Gym environment by wrapping the environment with envlogger.DMEnvLogger, which writes episodes to disk in real-time as TFRecord shards. For teams working outside the TensorFlow ecosystem, the rlds_creator tool provides command-line conversion from common formats, and the fog_x library offers a format-agnostic abstraction layer over RLDS and other robotics dataset formats.

Migrating an existing robotics dataset to RLDS involves three phases: schema mapping, episode segmentation, and builder implementation. In the schema mapping phase, you define a features dictionary that maps your existing data fields to RLDS conventions. Observations go into a nested dictionary keyed by modality (image, wrist_image, state), actions are a single array matching your robot's action space, and metadata like natural language instructions are stored as tf.string features. The key decision is whether to store images as raw tensors (faster loading, larger files) or JPEG-compressed bytes (smaller files, decode overhead).

Episode segmentation is the second critical step. If your data is already organized by episode (as in HDF5 group-per-episode layouts), this is straightforward. For continuous recordings like ROS bags, you need to define episode boundaries, typically at task completion, failure, or reset events. The Open X-Embodiment data standard recommends including the terminal observation (the observation after the last action) as a final step with is_last=True, which many downstream models rely on for value estimation.

Finally, you implement a TFDS DatasetBuilder. The builder's _info method returns the feature structure, _split_generators defines train/val/test splits, and _generate_examples yields episodes. Running tfds build generates the sharded TFRecords, dataset_info.json, and label metadata. The resulting dataset can be loaded with a single tfds.load() call, and the Open X-Embodiment project maintains a registry of compatible builders at github.com/google-deepmind/open_x_embodiment.

Claru delivers datasets in RLDS format with standard episode/step structure compatible with the Open X-Embodiment ecosystem. Feature connectors are configured for your specific observation and action spaces, with dataset_info.json metadata and optional TFDS builder code for seamless integration.

Every delivery includes a complete TFDS DatasetBuilder that can be registered in your local TFDS installation, enabling one-line loading with tfds.load('claru_your_dataset'). Image features are JPEG-compressed by default with configurable quality, and proprioceptive states use float32 precision. We validate schema compatibility with RT-X and Octo model expectations so your data is training-ready out of the box. For teams that need PyTorch compatibility, we include a dlimp-compatible wrapper and can additionally deliver the same data in LeRobot or HDF5 format at no extra cost.