Training Data for RT-1

Claru delivers RT-1-compatible datasets in RLDS format with 300x300 RGB observations, 256-bin discretized 7-DoF action labels, USE-encoded language instructions, and episode termination signals. Our data pipeline handles workspace-normalized action discretization, camera calibration, and the specific frame history format RT-1 expects. Since RT-1's data format is the foundation for the Open X-Embodiment ecosystem, Claru data also works directly with OpenVLA, Octo, RT-2-X, and any RLDS-compatible model. Our primary value-add is task diversity at scale: we can collect 50-200 demonstrations each across dozens of manipulation tasks simultaneously through our distributed network, matching RT-1's finding that breadth of task coverage drives generalization more than depth.

RT-1 (Robotics Transformer 1) is a large-scale robot learning model developed by Google's Everyday Robots team, published by Anthony Brohan, Noah Brown, Justice Carbajal, and over 45 co-authors in December 2022 (arXiv 2212.06817, presented at RSS 2023). RT-1 demonstrated for the first time that a single Transformer-based policy trained on a massive real-world dataset could perform over 700 instructions at a 97% success rate, while generalizing to novel tasks, distractors, and backgrounds significantly better than prior approaches.

The central contribution was not a novel architecture but a proof of scale: by collecting 130,000 teleoperated demonstrations across 744 distinct tasks over 17 months using a fleet of 13 Everyday Robots mobile manipulators, the team showed that real-world robot data at sufficient scale and diversity produces policies that generalize robustly. This was a paradigm-defining result -- prior work had been limited to hundreds or low thousands of demonstrations and a handful of tasks, with correspondingly narrow generalization.

RT-1's task repertoire covered a wide range of everyday manipulation behaviors: picking objects of various sizes and shapes, placing them in specific locations, opening and closing drawers, moving objects near other objects, upright placement, and basic navigation. The language-conditioned interface allowed operators to command the robot with natural language instructions like 'pick up the red can' or 'move the green chip bag near the orange,' and the model learned to ground these instructions in the visual scene and execute appropriate motor commands.

RT-1's impact extends well beyond its own results. It established the data format and collection methodology that became the foundation for the Open X-Embodiment project, which aggregated robot datasets from 22 institutions. RT-1's successor, RT-2, demonstrated that pre-trained vision-language models could be fine-tuned for robot control, and the RT-1 data format (later standardized as RLDS) became the de facto standard for robot learning datasets. Understanding RT-1's data requirements is therefore essential for anyone working with modern robot learning systems.

RT-1's architecture is a carefully engineered pipeline of three components: a FiLM-conditioned EfficientNet-B3 image encoder, a TokenLearner compression module, and a Transformer decoder. The total model size is only 35 million parameters -- remarkably small compared to subsequent VLA models like RT-2 (55B) or OpenVLA (7B) -- yet it achieves strong performance because the architecture was specifically optimized for real-time robot control at 3 Hz inference.

The image encoder is an EfficientNet-B3 backbone (26 MBConv blocks, 16M parameters) conditioned on language via Feature-wise Linear Modulation (FiLM). The natural language instruction is first encoded into a 512-dimensional embedding using Google's Universal Sentence Encoder, and this embedding is injected into each convolutional block of the EfficientNet via learned affine transformations. This produces 81 vision-language tokens per frame that capture both visual scene understanding and language-conditioned attention -- the model literally attends to different parts of the image depending on the instruction.

The TokenLearner module compresses these 81 tokens down to just 8 tokens per frame. This aggressive compression is essential for real-time operation: with 6 frames of image history, the uncompressed token sequence would be 486 tokens, but TokenLearner reduces this to 48 tokens that capture the most informative spatial features. These 48 tokens (with positional encoding) are fed into a decoder-only Transformer backbone that predicts discretized action tokens.

The action space discretization is a key design choice. Rather than predicting continuous action values, RT-1 discretizes each action dimension into 256 bins and treats action prediction as a classification problem. This avoids the mode-averaging problem that plagues continuous regression in multi-modal action distributions -- when there are multiple valid ways to grasp an object, regression averages them into an invalid action, while classification can represent each mode as a separate bin. The 7-DoF arm actions plus 3-DoF base actions plus 1 termination token yield 11 classification heads, each producing a softmax over 256 bins.

RT-1's training dataset is one of the most extensively documented robot learning datasets in the literature. The 130,000 demonstrations were collected over 17 months using 13 Everyday Robots mobile manipulators operating in office kitchen environments. Each demonstration was collected via teleoperation, with a human operator controlling the robot arm and base using a remote interface. The demonstrations span 744 distinct task instructions, averaging approximately 175 demonstrations per task, though the distribution is long-tailed -- common tasks like 'pick up the Coke can' have 500+ demonstrations while rare tasks may have as few as 50.

Each demonstration consists of: (1) A sequence of 300x300 RGB images from the robot's head-mounted camera at 3 Hz. (2) The corresponding 7-DoF end-effector delta actions (position + orientation + gripper), also at 3 Hz. (3) 3-DoF base velocity commands for the mobile base. (4) A natural language instruction string describing the task. (5) An episode termination signal indicating when the human operator judged the task complete. Typical episodes last 10-30 seconds (30-90 timesteps at 3 Hz).

A critical lesson from RT-1 is that task diversity matters more than per-task volume. The paper's ablation studies showed that training on 744 tasks with moderate demonstrations per task produced substantially better generalization than training on fewer tasks with more demonstrations each. Specifically, the model generalized 25% better to novel tasks, 36% better to novel distractors, and 18% better to novel backgrounds compared to the next best baseline. This finding has shaped all subsequent data collection strategies in the field.

The data collection methodology itself was a significant engineering achievement. The 13-robot fleet operated in parallel across multiple kitchen environments, with human operators working in shifts. Quality control included automatic filtering of demonstrations where the robot collided with objects, the task instruction was ambiguous, or the final state did not match the task specification. The team estimated that approximately 20% of raw demonstration attempts were discarded during quality control, meaning the effective collection effort was roughly 160,000 attempts to yield 130,000 clean demonstrations.

For teams replicating RT-1's data format, the key specifications are: 300x300 RGB images (center-cropped from the camera's native resolution), 3 Hz recording frequency, 256-bin discretization for each action dimension with per-dimension normalization to the robot's workspace bounds, and Universal Sentence Encoder embeddings for the language instructions. The data is stored in RLDS format (TFRecord files with standardized episode structure), which has become the community standard for robot learning datasets.

Claru delivers datasets in the exact format RT-1 and its derivatives (RT-2, RT-X, OpenVLA) consume. Our data pipeline outputs RLDS-formatted TFRecord files with 300x300 RGB observations, discretized 7-DoF action labels (256 bins per dimension with workspace-normalized bounds), natural language task instructions, and episode termination signals. For teams using RT-1-derived architectures, our data can be mixed directly into the training pipeline without format conversion.

While the RT-1 model weights are not publicly available, its data format lives on as the backbone of the Open X-Embodiment ecosystem. Claru provides RT-1-compatible data that is interoperable with OpenVLA, Octo, RT-2-X, and any model trained on RLDS-format datasets. This means data collected for RT-1 compatibility automatically works with the most widely used open-source robot learning models.

Claru's primary value-add for RT-1-style training is task diversity at scale. RT-1's key insight -- that 744 tasks with moderate per-task demonstrations outperforms deep coverage of few tasks -- requires a data collection operation that can efficiently parallelize across many tasks. Our distributed collection network, spanning 100+ cities with trained operators on multiple robot platforms, can deliver the breadth of task coverage that RT-1-style training demands. We can collect 50-200 demonstrations each for dozens of manipulation tasks simultaneously, rather than the serial approach of exhaustively collecting one task at a time.

For teams extending RT-1 to new robot platforms, Claru handles the action space mapping. We record raw end-effector poses and joint states, then discretize and normalize to the target action space specification. We provide complete calibration metadata including camera intrinsics, workspace bounds (used for action normalization), and robot URDF, enabling reproducible training and deployment.