Egocentric Kitchen Video Dataset

Kitchens are the primary target environment for household robotics. Companies building robotic kitchen assistants need training data that captures the full complexity of real cooking workflows — multi-step recipes, tool handling, ingredient manipulation, and the spatial reasoning required to navigate cluttered countertops. First-person video from real kitchens provides this signal in a way that simulation cannot replicate.

The egocentric viewpoint is critical because it matches the camera perspective of a robot operating in the same space. A head-mounted or chest-mounted camera captures exactly what a robot's onboard camera would see: hands reaching into drawers, objects partially occluded by other items, steam from cooking, and the dynamic lighting conditions of real kitchens. This perspective alignment dramatically reduces the domain gap between training data and deployment.

Kitchen environments present unique challenges for computer vision and robot learning. Transparent objects (glass bowls, measuring cups), deformable materials (dough, vegetables being chopped), liquids, and steam create visual complexity that standard object detection struggles with. Training on diverse real kitchen video is the most reliable path to robust perception in these conditions.

Claru collectors wear lightweight head-mounted cameras (GoPro Hero 12 or similar) while performing genuine cooking tasks in their own kitchens. This is not scripted — collectors follow actual recipes and perform real meal preparation, generating naturalistic data with authentic object interactions, timing, and error recovery behaviors that scripted collection cannot produce.

Each collection session captures 30-90 minutes of continuous activity. Collectors are recruited across geographic regions to ensure diversity in kitchen layouts (galley, L-shaped, island, commercial), equipment (gas vs electric, various appliance brands), and cooking styles (Western, Asian, South Asian, Latin American). This diversity is essential for training models that generalize beyond a single kitchen configuration.

Raw video is captured at 1080p or 4K resolution at 30fps, with optional depth sensing from RealSense D435 cameras for collectors who use our depth rig. Each session includes metadata: kitchen layout sketch, appliance inventory, and recipe or task description. This metadata enables researchers to filter and subset the data for specific training objectives.

VLA models like RT-2 and OpenVLA require diverse manipulation demonstrations to learn kitchen tasks. Claru's kitchen dataset provides the observation sequences these models need, with consistent action annotations that can be mapped to robot action spaces through retargeting. The diversity of kitchen layouts ensures the model does not overfit to a single environment.

World models trained on kitchen video learn the causal structure of cooking: what happens when you pour liquid into a hot pan, how dough deforms when kneaded, how steam disperses. These physical dynamics are critical for predictive models that plan multi-step cooking sequences. The temporal density of our annotations (every action boundary labeled) provides the supervision signal world models need.

Activity recognition and anticipation models benefit from the naturalistic collection protocol. Because collectors perform real cooking rather than scripted motions, the data includes genuine error recovery, multitasking (stirring while chopping), and realistic timing variation that synthetic or scripted datasets lack.