Training Data for RoboCat

Claru provides the two data assets that RoboCat-style self-improving systems cannot generate on their own: high-quality seed demonstrations and success classifier labels. Our seed demonstrations meet the strict quality bar the self-improvement loop requires — 100% successful completions, diverse object configurations, smooth motion profiles, and hardware-synchronized multi-view recording at 5-10 Hz. For the success classifier, our annotation team labels rollout outcomes with task-specific binary or graded success criteria, achieving 95%+ inter-annotator agreement. For cross-embodiment pipelines, Claru collects data across multiple robot platforms with consistent formatting while preserving each embodiment's native action space. All deliverables include full provenance: operator ID, session metadata, environment configuration, and per-episode quality scores.

RoboCat is a self-improving foundation agent for robot manipulation developed by Google DeepMind, introduced in June 2023. The core innovation is a training loop where the model generates its own training data through autonomous practice, filters that data for successful task completions, and retrains on the expanded dataset — progressively improving without additional human demonstrations. RoboCat was the first system to demonstrate that a single vision-language-action model could autonomously improve across multiple robot embodiments and hundreds of tasks.

The system builds on the Gato architecture — a multimodal transformer that processes tokenized images, text, and actions in a unified sequence. RoboCat extends Gato with a structured self-improvement protocol: starting from a small seed dataset of human demonstrations (as few as 100 per task), the model performs autonomous rollouts in the real world, a success classifier filters the rollouts for task completions, and the successful trajectories are added to the training set. The model is then retrained from scratch on the expanded dataset. Each iteration of this loop yields 2-3x improvement in task success rate.

DeepMind demonstrated RoboCat across four different robot embodiments: a Sawyer arm, a Panda arm, a KUKA arm, and a custom bi-manual setup. The model was trained on 253 tasks spanning pick-and-place, stacking, insertion, lid manipulation, and more complex assembly operations. Critically, the cross-embodiment training was not just a curiosity — demonstrations from one robot platform measurably improved performance on other platforms, even when the embodiments had different kinematic structures, grippers, and camera configurations.

RoboCat's approach has important implications for data strategy. Because the model generates and curates its own data after the initial bootstrapping phase, the quality and diversity of the seed demonstrations is paramount. Poor seed data leads to a negative spiral where the model's autonomous practice generates low-quality trajectories that further degrade performance. Conversely, high-quality seed demonstrations enable a virtuous cycle where each iteration compounds improvements.

RoboCat's architecture is an extension of DeepMind's Gato model — a 1.2 billion parameter multimodal transformer that tokenizes all inputs and outputs into a single sequence. Visual observations (multi-view RGB images) are processed through a Vision Transformer (ViT) encoder that produces a fixed number of visual tokens per frame. These tokens are interleaved with tokenized proprioceptive state (joint positions and velocities), language tokens (the task description), and action tokens in an autoregressive sequence. The model is trained to predict the next action token given the preceding context.

The key architectural innovation is not in the model itself but in the self-improvement training loop. The protocol works as follows: (1) Collect an initial seed dataset of 100+ human demonstrations per task. (2) Train the model on this seed data plus any existing cross-embodiment data. (3) Deploy the trained model for autonomous practice — the robot attempts the task repeatedly without human intervention. (4) Run a trained success classifier on the autonomous rollouts to identify successful completions. (5) Add successful rollouts to the training dataset. (6) Retrain the model from scratch on the expanded dataset. Steps 3-6 repeat for 5+ iterations.

Cross-embodiment transfer is achieved through the tokenized representation. Because all embodiments share the same tokenization scheme (actions are discretized into 1024 bins per dimension, regardless of the underlying hardware), the model learns a shared policy representation across platforms. The paper shows that training on data from all four embodiments jointly outperforms training on data from any single embodiment, even for that embodiment's own tasks. This suggests the model transfers not just visual understanding but also manipulation strategies across kinematically different robots.

The success classifier that filters autonomous rollouts is itself a critical component. DeepMind trained a separate vision-based classifier per task that evaluates the final state of each rollout. The accuracy of this classifier directly affects the quality of the self-generated data — false positives inject failed trajectories into the training set, while false negatives discard useful data. The paper reports classifier accuracies of 85-95% depending on task difficulty, with harder tasks (multi-step assembly) showing lower accuracy.

RoboCat's data requirements are unique because of the self-improvement loop. The model's training dataset grows over time, but the initial seed data determines whether the self-improvement process converges to high performance or collapses. The paper uses 100 human demonstrations per task as the minimum seed, collected via teleoperation on each of the four robot platforms. Each demonstration consists of multi-view RGB video (overhead and wrist cameras), proprioceptive state at 5-10 Hz, end-effector actions, and a natural language task description.

Seed demonstration quality is the single most important factor. The 100 demonstrations must be: (1) Successful — every trajectory should complete the task to the goal state, with no failed or partial attempts in the seed set. (2) Diverse — the demonstrations should cover the natural variation in object positions, orientations, sizes, and colors that the robot will encounter during autonomous practice. (3) Smooth — trajectories should be continuous and efficient, without unnecessary pauses, restarts, or jittery movements that would confuse the model. (4) Consistently formatted — all demonstrations for a given embodiment must share the same camera viewpoints, action space normalization, and control frequency.

For cross-embodiment training, the paper shows that pooling data across all four platforms — even with different kinematic structures — consistently improves per-platform performance. This means teams building RoboCat-style systems should collect seed data from as many robot platforms as possible, even if the end goal is deployment on a single platform. The shared visual and language understanding transfers across embodiments, providing a stronger starting point for the self-improvement iterations.

After the initial seed phase, the success classifier becomes a critical data quality gate. The classifier itself requires training data: typically 500-1,000 labeled rollout outcomes (success/failure) per task, annotated by human operators. This annotation cost is often overlooked but is essential for the self-improvement loop to function. Poor classifier accuracy leads to either dataset poisoning (false positives) or data starvation (false negatives), both of which degrade the improvement loop.

Claru provides the high-quality seed demonstrations that RoboCat-style self-improving systems require to bootstrap the autonomous improvement loop. Our operators are trained specifically for seed-quality collection: every trajectory is a successful completion, covers the full diversity of object configurations, follows smooth and efficient motion profiles, and is recorded with hardware-synchronized multi-view cameras at the control frequency your embodiment requires (5-10 Hz for RoboCat-compatible pipelines).

Beyond seed demonstrations, Claru provides the human annotations needed to train the success classifier that gates the self-improvement loop. Our annotation team labels rollout outcomes (success/failure) with task-specific criteria, achieving inter-annotator agreement rates above 95%. We can also provide graded success labels (partial completion scores) for tasks where binary success/failure is too coarse — for example, multi-step assembly tasks where the robot may complete 3 of 5 steps.

For teams building cross-embodiment datasets, Claru collects demonstrations across multiple robot platforms with consistent data formatting. We normalize camera placement, resolution, and frame rate across embodiments while preserving each platform's native action space — matching RoboCat's requirement for hardware-specific action spaces within a shared visual/language representation. All datasets include full provenance documentation: operator ID, collection session, environment configuration, and per-episode quality scores.