How to Collect Kitchen Activity Data for AI Training

Kitchen activity datasets demand a well-defined taxonomy before any camera is turned on. Start by selecting your target granularity: coarse-grained (meal-level labels like 'making breakfast'), medium-grained (recipe-level like 'scrambling eggs'), or fine-grained (action-level like 'crack egg into bowl'). For most embodied AI applications, fine-grained is necessary. Adopt the EPIC-KITCHENS verb-noun vocabulary as your baseline — its 97 verbs and 300 nouns cover the vast majority of kitchen activities and enable direct benchmarking against published results.

Create a recording protocol document specifying: target recipe list (15-30 recipes spanning breakfast, lunch, dinner, and snack categories), required ingredient and tool variations per recipe (e.g., 'cut tomato' must be recorded with both a chef's knife and a serrated knife, on both a cutting board and a plate), participant screening criteria (must be comfortable cooking the target recipes without instructions — scripted cooking produces unnaturally stilted movements), session structure (1 camera check, 2-3 meal recordings per session, each 15-45 minutes), and environment documentation requirements (photograph the kitchen layout, measure counter heights, record ambient light levels with a lux meter). For egocentric collection, specify the camera mounting position (forehead center, 15-degree downward tilt) and calibration procedure (look at 5 fiducial markers at known positions). Ship the protocol to 3 pilot participants and iterate based on issues discovered.

Whether using a lab kitchen or real homes, sensor installation follows the same principles. For fixed cameras: mount using articulating arms with adhesive bases (no drilling in participants' homes) at positions covering the stove, prep area, sink, and refrigerator. Each camera should capture at least 60% overlap with an adjacent camera to enable multi-view reconstruction. Set all cameras to manual exposure (1/60s shutter, ISO 400-800 depending on lighting) to avoid auto-exposure fluctuations when flames or oven lights toggle. Synchronize cameras using PTP or by recording a visual sync clapboard at the start of each session — audio sync from the clap peaks achieves sub-frame accuracy.

For egocentric collection: configure GoPro cameras in Linear lens mode (reduces fisheye distortion), 1080p at 30fps (4K generates 300+ GB per session and rarely improves downstream model performance), with Protune enabled for flat color profile (better for post-processing). If using Meta Aria glasses, enable all sensor streams: RGB (1408x1408 at 30fps), stereo monochrome (640x480 at 30fps), IMU (1000 Hz), eye tracking (30 Hz), and hand tracking (30 Hz). The eye-tracking data is particularly valuable — it reveals which objects the participant is attending to, providing a natural curriculum signal for robot learning. Verify storage capacity: a 3-hour collection session generates 50-80 GB for a multi-camera fixed setup or 30-50 GB per egocentric rig. Bring spare SD cards and portable SSDs. Test the full pipeline with a 15-minute dry run before each participant arrives.

Recruit 20-50 participants across a range of cooking skill levels, ages (18-65+), handedness, and cultural backgrounds to ensure recipe and technique diversity. Screen for cooking competence on your target recipes — participants should be able to cook without written instructions, as reading recipes during recording creates unnatural pauses and head-down posture that occludes the workspace in egocentric video. Compensate participants at $30-50/hour to reflect the effort of cooking 2-3 full meals per session.

Structure each collection session as: (1) Consent and briefing (15 min) — explain the project, collect signed consent forms, review the recipe list. (2) Kitchen documentation (10 min) — photograph layout, measure workspace dimensions, note appliance brands and locations. (3) Sensor setup and calibration (15-20 min) — mount cameras, run sync, verify recording. (4) Warm-up recipe (15-20 min, not included in dataset) — participant makes a simple dish to get comfortable with the cameras. (5) Target recordings (2-3 recipes, 15-45 min each) — participant cooks naturally while a research assistant notes any equipment issues. (6) Post-session review (10 min) — quick playback to verify video quality, participant debriefs on any unusual steps. Schedule sessions across different times of day to capture varying natural light conditions. In real homes, visit each kitchen at least twice (different meal types) to amortize the setup cost and increase per-kitchen diversity.

Temporal annotation is the most labor-intensive step. For fine-grained action segments, each minute of raw video typically requires 8-15 minutes of annotation time. Use ELAN for dense temporal annotation (it handles multi-tier hierarchical labels natively) or CVAT for combined spatial-temporal annotation (bounding boxes + activity labels). Set up the annotation schema with three parallel tiers: Tier 1 (coarse) — recipe phase labels spanning 1-10 minutes (prep, cook, plate, clean). Tier 2 (fine) — verb-noun action segments spanning 1-30 seconds (cut_tomato, pour_oil, stir_sauce). Tier 3 (atomic) — optional contact/manipulation primitives spanning 0.2-2 seconds (reach, grasp, transport, release).

Train annotators in a 2-day bootcamp covering: the taxonomy with 50+ visual examples per action class, boundary rules (action starts at first intentional motion toward the target object, ends when the hand releases or the tool disengages), and edge cases (simultaneous bimanual actions get separate annotations per hand, background monitoring like watching a pot counts as 'check' not 'stir'). Require each annotator to pass a qualification test: annotate 3 reference videos and achieve Cohen's kappa > 0.75 against the gold standard. For the production dataset, double-annotate 20% of videos and resolve disagreements through adjudication. Use inter-annotator agreement metrics to identify ambiguous action classes — if two classes consistently confuse annotators (e.g., 'mix' vs 'stir'), either merge them or add clearer visual criteria to the guidelines.

After annotation, build a feature extraction pipeline that converts raw video and labels into training-ready tensors. For each annotated action segment, extract: RGB frames at the target model's resolution (typically 224x224 or 256x256), optical flow fields using RAFT (state-of-the-art accuracy) or TV-L1 (faster, slightly lower quality) between consecutive frames, hand bounding boxes and poses using MediaPipe Hands or FrankMocap, and object bounding boxes using a DINO-v2 or Grounding-DINO detector fine-tuned on kitchen objects. Store extracted features in HDF5 with one file per session containing groups for each modality.

Run comprehensive validation: (1) Annotation coverage — verify that >98% of video frames have at least one active annotation (gaps indicate annotator errors or genuinely idle periods that should be labeled as 'idle'). (2) Temporal consistency — check that no two annotations of the same tier overlap for the same hand (physical constraint violation). (3) Class distribution — compute the frequency of each verb-noun pair and flag rare combinations (<5 occurrences) that may need additional collection or merging with a parent class. (4) Visual quality — sample 100 random segments and verify: no camera occlusion from steam or grease splatter, no motion blur exceeding 10% of frames, and correct participant de-identification. (5) Cross-kitchen statistics — compute per-kitchen action distributions and verify no single kitchen contributes more than 10% of any action class. Generate a dataset statistics report with histograms, confusion matrices from inter-annotator agreement, and qualitative examples of each action class.

Convert the validated dataset into standard formats for activity recognition and/or robot learning. For video understanding models (SlowFast, Video-MAE, TimeSFormer), the standard format is a directory of video clips (one per action segment, trimmed with 0.5s context before and after the annotated boundaries) plus a CSV manifest mapping clip paths to labels. For Ego4D-compatible formatting, follow the Ego4D annotation JSON schema with video_uid, clip_uid, verb_label, noun_label, start_frame, and end_frame fields. For robot learning from human video (using models like R3M or VIP), output the full uncut episodes as HDF5 files with synchronized observation streams (images, depth if available, hand pose) and action labels at each frame.

Generate train/validation/test splits stratified by participant AND kitchen — never allow the same participant or kitchen to appear in both train and test, as models can overfit to kitchen-specific visual features (that distinctive red KitchenAid mixer) or participant-specific behavior patterns. A standard split is 70/15/15 by participant count. For temporal splits, some benchmarks additionally ensure that test videos are recorded on different days than training videos. Compute split-level statistics and verify that action class distributions are similar across splits using Jensen-Shannon divergence (target JSD < 0.05). Package the dataset with: the raw video files, extracted features, annotation files, split definitions, a dataloader compatible with PyTorch and the standard video understanding libraries (Decord for video loading, pytorchvideo for transforms), and a comprehensive dataset card documenting collection conditions, participant demographics, annotation guidelines, and known limitations.