Object Manipulation Training Data

Claru operates a distributed network of 10,000+ trained data collectors across 100+ cities, enabling rapid collection of diverse object manipulation demonstrations at scale. Our infrastructure supports multi-view synchronized recording at 30+ Hz with hardware-triggered camera alignment achieving sub-1 ms synchronization, delivering data in RLDS, HDF5, zarr, or WebDataset format. With 386,000+ annotated manipulation clips already in our catalog spanning egocentric video, tabletop manipulation, and multi-environment recordings, we can supplement custom collections with pre-existing data to accelerate your training pipeline. Each demonstration is scored for trajectory quality, temporal synchronization, and task completion, with automated flagging and 25% human spot-verification. We support single-task datasets (50-500 demonstrations, 1-2 week turnaround) through foundation model-scale datasets (100K+ demonstrations, multi-month campaigns with parallel collection sites).

Object manipulation is the foundational capability for nearly every useful robot. Whether a robot is sorting packages in a warehouse, loading a dishwasher, or assembling electronics, it must grasp, transport, and place physical objects with precision and reliability. Training policies for these tasks requires large-scale, diverse demonstration data that captures the full distribution of object geometries, surface properties, lighting conditions, and scene configurations a robot will encounter in deployment.

The challenge is not merely volume but coverage. A manipulation policy trained on 10,000 demonstrations of picking up cubes will fail on cylinders. Research from Google DeepMind's RT-2 (Brohan et al., 2023) showed that scaling to 130,000 demonstrations across hundreds of object categories was necessary to achieve robust generalization. The Open X-Embodiment project (O'Neill et al., 2024) further demonstrated that cross-embodiment data from multiple robot platforms improves manipulation success rates by 50% compared to single-robot datasets. These findings establish that both object diversity and embodiment diversity are necessary for general-purpose manipulation.

Real-world deployment demands data collected in physical environments, not just simulation. While sim-to-real transfer has improved with domain randomization techniques (Tobin et al., 2017), policies trained exclusively on synthetic data still exhibit a 15-30% performance gap on novel objects compared to those trained on real-world demonstrations (James et al., 2019). This gap is especially pronounced for deformable objects, transparent surfaces, and cluttered scenes where physics engines produce unrealistic contact dynamics. The gap persists even with state-of-the-art rendering: transparent objects (bottles, glasses) cause depth sensor failures that simulators do not reproduce, and deformable objects (bags, cables, cloth) have contact physics too complex for real-time simulation.

The economics of manipulation data collection have shifted dramatically with the rise of foundation models. A team building a single-task policy (e.g., 'pick up a specific part from a tray') needs 50-500 demonstrations and can collect them in-house in a few days. A team fine-tuning a foundation model (OpenVLA, Octo, Pi-zero) for a new domain needs 5,000-50,000 demonstrations spanning diverse tasks and objects — a multi-week or multi-month collection effort requiring professional infrastructure. And a team building a proprietary foundation model from scratch needs 100,000-1,000,000+ demonstrations, which is infeasible without distributed collection across many sites and operators. At each scale, the data quality requirements are different: single-task data must be precise but narrow, while foundation model data must be diverse even at the cost of individual demonstration perfection.

Teleoperation remains the gold standard for collecting manipulation demonstrations. Leader-follower systems like ALOHA (Zhao et al., 2023) enable bimanual data collection at 50 Hz with sub-millimeter positional accuracy. VR-based teleoperation with devices like Meta Quest 3 offers lower hardware costs but introduces latency and kinematic mismatch that can degrade demonstration quality. The choice of interface directly impacts the downstream policy — ACT (Zhao et al., 2023) achieved a 20% higher success rate when trained on leader-follower data compared to VR-collected demonstrations on the same tasks.

Kinesthetic teaching, where a human physically guides the robot, provides natural demonstrations but limits data throughput to roughly 5 demonstrations per hour. In contrast, experienced teleoperators can produce 20-40 demonstrations per hour on tabletop tasks. For large-scale data collection, parallelized teleoperation stations with multiple operators collecting simultaneously can reach throughputs of 200+ demonstrations per day. The DROID project demonstrated this at scale: 90 institutions collecting in parallel produced 76,000 demonstrations in months rather than the years a single lab would require.

Autonomous data collection through scripted policies or reinforcement learning-guided exploration has emerged as a complement to human demonstrations. Google's RT-1 paper (Brohan et al., 2022) used a fleet of 13 robots operating autonomously for 17 months to collect 130,000 demonstrations. However, the resulting data distribution is biased toward already-learned behaviors, making human-collected edge cases essential for pushing policy boundaries. The practical approach is a hybrid: use autonomous collection for high-volume coverage of common manipulation scenarios, and human teleoperation for edge cases, novel objects, and failure recovery demonstrations.

Data quality varies significantly by collection method and has measurable impact on downstream policy performance. Mandlekar et al. (2021) showed that filtering the bottom 20% of demonstrations by trajectory smoothness and task completion time improved Diffusion Policy success rates by 15% compared to training on unfiltered data. This finding has led to the practice of per-demonstration quality scoring: each demonstration is assigned scores for trajectory smoothness (jerk norm), task efficiency (completion time relative to expert baseline), grasp quality (stability after lift), and overall success. Low-scoring demonstrations are either excluded or downweighted during training.

High-quality manipulation data requires precise temporal synchronization across all sensor streams. Camera timestamps must be aligned within 5 ms of proprioceptive readings to prevent the policy from learning incorrect visual-action correspondences. At typical manipulation velocities (0.3-1.0 m/s), a 10 ms desynchronization means the visual observation and the corresponding action are spatially offset by 3-10 mm — enough to degrade fine manipulation performance. Claru's data collection infrastructure uses hardware-triggered synchronization to guarantee sub-millisecond alignment across up to 8 simultaneous camera streams, with verification at the start of each collection session.

Annotation standards for manipulation data extend beyond simple task success labels. Rich annotations include grasp type classification (power, pinch, lateral, precision), contact event timestamps (first-contact, stable-grasp, lift-off, pre-place, release), object state changes (picked, transported, placed, stacked, inserted), and failure mode categorization (slip, collision, timeout, wrong object, partial completion). These structured annotations enable filtering and curriculum learning strategies that can improve training efficiency by 30-40% compared to naive random sampling (Mandlekar et al., 2021). For language-conditioned models, each demonstration additionally requires a natural language task description that is both accurate and varied — templated descriptions like 'pick up the red cube' produce worse language grounding than diverse phrasings like 'grab that red block,' 'get the crimson cube,' 'pick up the small red box.'

Data diversity is quantified along multiple axes: object geometry (convex, concave, articulated), material properties (rigid, deformable, granular), scene complexity (isolated, cluttered, occluded), and task variation (pick-place, stack, insert, pour, open, close). A production-grade manipulation dataset should cover at least 100 distinct object instances across 10+ material categories, collected in 5+ distinct environment configurations. The object set must include challenging categories that trip up vision systems: transparent objects (glass, clear plastic), reflective objects (metal, mirrors), thin objects (cards, papers), and small objects (screws, pills). Claru datasets routinely exceed these thresholds with data from over 100 cities worldwide.

Object diversity follows a power law in real-world deployment: 80% of manipulation acts involve 20% of object categories (common items like boxes, bottles, cups), while the remaining 80% of categories appear rarely but critically. Training data must cover the long tail to prevent catastrophic failures on uncommon objects. Claru's object sampling protocol allocates 60% of collection episodes to common objects (ensuring deep coverage) and 40% to long-tail objects (ensuring breadth), with the long-tail set rotated across collection sessions to maximize unique object exposure without sacrificing per-object repetition count.