Training Data for Toyota Research Institute

TRI occupies a unique position in the robotics landscape — backed by Toyota's resources and strategic interest in aging-society solutions, they combine cutting-edge AI research with a clear deployment target: household assistive robots. Their diffusion policy work (Chi et al., 2023) has become one of the most influential approaches in robot learning, and their continued development of this framework drives specific data requirements.

Diffusion policy excels at contact-rich manipulation precisely because it can model multi-modal action distributions — when there are multiple valid ways to fold a cloth or load a dishwasher, the diffusion model captures this distribution rather than averaging across modes. But this capability demands training data that contains diverse solutions to the same task. For cloth folding alone, TRI needs demonstrations showing different folding strategies, cloth types, surface conditions, and starting configurations.

The deformable object challenge is particularly data-hungry. Deformable objects — cloth, food items, flexible packaging, cables — have effectively infinite state spaces that cannot be exhaustively explored in simulation. While simulators can model simple cloth physics, the interaction between real fabric, real surfaces, and real grippers involves friction, material compliance, and draping dynamics that differ dramatically from simulation. Real-world demonstrations of deformable manipulation are irreplaceable.

TRI's assistive robotics vision adds a demographic dimension to the data requirement. Robots that assist elderly individuals must understand home environments as configured by actual residents — not standardized laboratory kitchens. Medicine bottles in bathroom cabinets, groceries in varying refrigerator layouts, clothing in diverse closet configurations. Claru's ability to collect data in real homes across many locations provides the environmental authenticity that TRI's assistive vision demands.

The ALOHA bimanual system, originally developed at Stanford and adopted extensively by TRI, has become the standard hardware for scalable bimanual data collection. ALOHA's low cost (~$20K per setup) makes it practical to deploy multiple collection stations, but the data diversity is still limited by the environments where stations are placed. TRI's adoption of ALOHA-style systems creates a natural partnership with Claru's distributed collection model — deploying low-cost teleoperation rigs across Claru's collector network to generate bimanual demonstration data from diverse real-world settings.

TRI's recent Track2Act work demonstrates learning manipulation from internet videos by predicting point tracks — visual correspondences that show how objects move during manipulation. This approach can leverage Claru's large-scale egocentric video as a pretraining resource, since first-person cooking and household videos naturally contain the dense manipulation observations that point-track models need.