How to Collect Egocentric Video Data for AI Training

Choose a camera and mounting configuration based on your target use case. For general activity capture: GoPro Hero (latest generation) with SuperView mode (ultra-wide FOV) mounted on a head strap. Head mounting captures gaze direction implicitly and closely matches a humanoid robot's head camera viewpoint. For hand manipulation detail: mount the camera on a chest harness, angled slightly downward to center on the hand workspace. Chest mounting is more stable than head mounting and captures hand-object interactions with less motion blur.

Record at 1080p 30fps with linear FOV correction enabled (reduces barrel distortion from wide-angle lenses). Enable electronic image stabilization (EIS) to reduce motion blur from head/body movements. Set white balance to auto and exposure to auto — this introduces visual variation that improves model robustness to different lighting conditions. Prepare 3-5 identical camera setups for parallel collection by multiple participants.

Define what activities should be captured, in which environments, and with what variation. Create an activity menu — a list of 20-50 activities organized by category (kitchen tasks, office tasks, workshop tasks, outdoor tasks). For each activity, specify: the goal (what the person should accomplish), the expected duration (30s to 10min), required objects, and the environment type.

Design collection sessions of 2-3 hours. Each session should cover 8-15 different activities in 2-3 different environment configurations. Provide the participant with the activity menu but let them execute tasks naturally — do not script the exact sequence of motions. After each activity, have the participant verbally narrate what they did (recorded by the camera's microphone) for later transcription and annotation. Include transition periods (walking between rooms, setting up the next task) in the recording — these provide navigation data that is also useful for training.

Recruit 20-50 participants across demographics (age, gender, handedness) and skill levels for your target activities. Obtain informed consent covering: video recording, storage, AI training use, privacy processing, and data retention period. Have participants sign a consent form before any recording begins.

Train participants on the protocol in a 30-minute onboarding session. Cover: how to wear and adjust the camera, how to start/stop recording, the activity menu and expectations, how to narrate activities, and when to pause for privacy (entering bathrooms, encountering people who have not consented). Run a 15-minute practice recording and review it with the participant to verify camera positioning, audio quality, and activity coverage. Participants who cannot follow the protocol or produce consistently poor-quality footage should be replaced rather than retained.

Run the collection campaign with daily quality monitoring. At the end of each collection day, spot-check 10-15% of recorded footage for: camera positioning (centered on the workspace, not pointing at the ceiling), image quality (in focus, properly exposed, minimal motion blur), audio quality (narrations audible, minimal background noise), activity coverage (all planned activities were captured), and environment variation (different rooms, lighting conditions, object arrangements).

Track metadata per recording session: participant ID, location, date/time, activities covered, total recording duration, and any issues (battery died, camera fell, privacy incident). Maintain a progress dashboard showing: total hours collected, hours per environment type, hours per activity category, and participant utilization. Identify coverage gaps (underrepresented activities or environments) early and adjust the collection schedule to fill them.

Run the privacy and quality processing pipeline on all collected footage. Phase 1: Face detection and blurring — run a face detector (RetinaFace or YOLO-Face) on all frames and apply Gaussian blur to detected face regions. Phase 2: Screen and document detection — detect computer screens, phone screens, and paper documents that might contain personal information and blur them. Phase 3: Quality filtering — compute per-frame quality scores based on blur magnitude (optical flow variance), exposure (histogram analysis), and camera motion (too stationary = waiting, too jerky = running).

Generate a quality report per video: percentage of frames with detected faces (should decrease to near-zero after blurring), percentage of low-quality frames (target <15%), total usable duration, and flagged segments requiring manual review. Videos with >30% low-quality frames or unresolvable privacy issues should be discarded rather than patched.

Segment the continuous video recordings into clips based on activity boundaries. Use a combination of automatic activity boundary detection (based on motion patterns and audio cues) and manual verification. Each clip should contain one complete activity from start to finish, typically 30 seconds to 10 minutes.

Annotate clips with: activity labels (from a predefined taxonomy), temporal action segments (start/end timestamps for each sub-action), object labels (bounding boxes or just text labels for objects involved), natural language descriptions (what the person is doing and why), and environment metadata (location type, lighting condition, clutter level). Use the participant's verbal narrations as a starting point for natural language descriptions, then have annotators refine and expand them.

Package the dataset with a standard structure: one directory per clip containing the video file, annotation JSON, and metadata. Include a dataset card documenting the collection protocol, participant demographics, annotation schema, privacy processing details, and known limitations. Provide loading scripts for common training frameworks.