Food Preparation Training Data

Claru collects food preparation data in fully equipped kitchen stations with multi-camera rigs, calibrated weight scales, IR and probe thermometers, and organized ingredient libraries. Our recipe-card collection protocol ensures structured coverage of cooking primitives (cutting, mixing, pouring, heating, plating) across ingredient substitutions. Each episode includes recipe step timestamps, food state transition labels, ingredient identification tags, technique classifications, and final dish quality assessments. We support single-step primitive training and full multi-step recipe execution data across test kitchens, restaurant kitchens, and home environments. Datasets are delivered with full sensor calibration, ingredient metadata, and recipe-stratified train/val/test splits in RLDS, HDF5, or custom formats.

Robotic food preparation spans the full cooking pipeline: retrieving ingredients, measuring quantities, cutting and slicing, mixing and stirring, applying heat, seasoning, and plating. It is arguably the most demanding manipulation domain because it combines nearly every robotics challenge in a single workflow: deformable object manipulation (dough, vegetables), liquid handling (pouring, stirring), tool use (knives, spatulas), thermal management (not burning food), and long-horizon sequential planning (following a recipe with 10-30 steps).

The data challenge is compounded by irreversibility. Most manipulation tasks are retry-friendly — a failed grasp can be reattempted. In cooking, many operations are one-way: over-chopped vegetables cannot be reassembled, burnt sauce cannot be unburnt, and seasoning cannot be removed. This means training data must capture not just successful executions but the perceptual cues that signal when to stop — the color change of sauteed onions, the consistency of whipped cream, the sound of sizzling oil reaching temperature. These multi-modal signals require sensor stacks beyond standard RGB cameras.

Commercial interest is substantial. Companies like Moley Robotics, Dexai Robotics, Miso Robotics (Flippy), and Samsung (Bot Chef) are building kitchen automation systems. The restaurant industry alone spends $350B annually on labor in the US, and fast-food chains see kitchen robots as critical for staffing-challenged locations. However, each recipe variant requires dedicated training data, and a single restaurant menu with 30 items may need 50,000+ demonstrations to cover the ingredient and technique diversity of production cooking.

The safety dimension in food preparation is unique to this domain. Robots handling sharp knives, hot pans, and open flames operate in a workspace that is hazardous to both the robot and nearby humans. Cutting demonstrations must capture safe knife handling practices: blade orientation during transport, cutting board stabilization, and finger-safe grip positions. Heat-related demonstrations must include safe distance maintenance from open flames and hot surfaces, pan handle orientation to avoid tipping, and monitoring of oil temperature to prevent flash fires. These safety-critical behaviors must be explicitly captured and annotated in training data — they are rarely the focus of academic datasets but are non-negotiable for commercial deployment.

The ALOHA system (Zhao et al., 2023) demonstrated bimanual food preparation from teleoperation — including stir-frying shrimp and flipping pancakes — but required dedicated training per dish. Mobile ALOHA (Fu et al., 2024) extended this to include navigation (moving between counter and stove) with 50 demonstrations per task. SayCan (Ahn et al., 2022) showed language-conditioned kitchen task execution but was limited to pick-and-place operations, not actual cooking.

The emerging paradigm is recipe-grounded manipulation. Rather than training one policy per dish, a recipe-conditioned policy takes natural language recipe steps as input and executes the corresponding manipulation. This requires paired (instruction, demonstration) data where each recipe step is aligned with the video segment showing its execution. The EPIC-KITCHENS dataset (Damen et al., 2022) provides 100 hours of egocentric kitchen video with narration, but these are human recordings without robot action labels — a gap between observation and manipulation data.

Food state tracking is the critical perception challenge. A policy must detect when dough has been kneaded enough (elasticity visual cues), when vegetables are sauteed (color change from green to golden), and when liquid has reduced to the right consistency (viscosity from stirring resistance). These are learned perceptual concepts that require large annotated datasets. Current datasets annotate food state transitions coarsely (raw/cooking/done); production systems need continuous state estimation with quantitative metrics (internal temperature, moisture content, texture score).

Multi-step reasoning presents the most complex planning challenge. A recipe like stir-fry involves parallel execution: heating oil while cutting vegetables, monitoring rice while sauteing the main dish, and timing sauce reduction to finish simultaneously with the protein. Demonstrations of multi-step recipes must capture not just individual technique execution but the operator's temporal planning — when they choose to start heating a pan, how they interleave cutting with monitoring, and how they adjust timing when an ingredient takes longer than expected. This temporal reasoning data, annotated with parallel task state and timing decisions, is essential for training policies that can manage the concurrent workflows of real cooking.

Food preparation collection requires a fully equipped kitchen station: countertop workspace with cutting boards and prep surfaces, cooking stations (stovetop, oven access), a sink for washing, and organized ingredient storage. The sensor stack includes overhead RGB-D cameras for workspace coverage, side-view cameras for tool use visibility, wrist cameras for close-up food state observation, weight scales for portioning, and temperature sensors (IR thermometer and probe thermometer) for cooking tasks.

Each collection session follows a recipe card that specifies ingredients, quantities, steps, and target outcomes. The operator executes the recipe via teleoperation while a recipe annotator timestamps each step transition. Post-hoc annotations add food state labels at key transitions (raw, chopping, chopped, cooking, seared, plated), ingredient identification tags for each item in frame, technique classification (julienne, dice, mince, fold, whip, saute), and quality assessments for the final dish (appearance, portioning accuracy, completion of all steps).

Ingredient diversity is managed through a structured ingredient matrix. Each recipe is collected with 3-5 ingredient substitutions (e.g., chicken versus tofu, broccoli versus green beans) to prevent the policy from overfitting to specific ingredients. Claru's food preparation protocol also includes deliberate variation in ingredient presentation — pre-washed versus unwashed vegetables, different cut sizes in raw ingredients, varying initial workspace organization — to build robustness to real kitchen conditions.

Safety annotations form a mandatory layer in Claru's food preparation data. Every episode involving a knife includes blade-orientation-during-transport labels, grip safety assessments, and cutting board stability checks. Heat-related episodes include pan handle orientation, safe distance compliance, and temperature hazard flags. These safety annotations enable training policies that do not just execute the cooking technique correctly but do so while maintaining safe operating practices — a requirement for any commercial food preparation robot that operates near human workers or in consumer-accessible spaces.