How to Build an Egocentric Video Data Pipeline

Map out the end-to-end pipeline from raw video ingestion to enriched dataset delivery. An egocentric video pipeline has four stages: ingest (receive raw video files, validate, and archive), process (extract frames, run quality checks, apply privacy filters), enrich (run vision models for depth, segmentation, activity recognition, and captioning), and deliver (package enriched data in the target format and push to the consumer).

Define a data schema that tracks every artifact. Each raw video clip gets a unique clip_id. The schema should include: clip_id (UUID), source_camera (gopro_hero12, aria, realsense), collector_id, recording_date, duration_seconds, resolution, fps, location_type (kitchen, office, workshop, outdoor), activity_category, raw_s3_key, privacy_status (pending, blurred, reviewed), and processing_status (ingested, frames_extracted, depth_complete, segmentation_complete, captions_complete, delivered). Store this metadata in a PostgreSQL database (or Supabase) that serves as the pipeline's single source of truth.

Choose an orchestration framework. For pipelines processing under 500 clips per week, a simple Python script with sequential processing and checkpointing is sufficient. For larger volumes, use Apache Airflow with a DAG that defines dependencies between stages: ingestion triggers frame extraction, which triggers parallel depth and segmentation jobs, which trigger captioning once both complete. Configure Airflow with a Celery executor backed by Redis for task queuing and an SQS or Pub/Sub trigger for new uploads. Define retry logic: depth estimation retries up to 3 times with exponential backoff, captioning retries twice, and any clip that fails after retries is flagged for manual review rather than blocking the pipeline.

The ingestion layer receives raw video files from collectors, validates them, and prepares them for processing. Set up an S3 bucket (or GCS bucket) with a standardized upload path: s3://your-bucket/raw/{collector_id}/{date}/{clip_id}.mp4. Configure an S3 event notification or Lambda trigger that fires when a new .mp4 file lands, registers the clip in the metadata database, and enqueues it for processing.

Frame extraction is the most storage-intensive stage. Use FFmpeg to decode video to individual frames: ffmpeg -i input.mp4 -vf 'fps=30' -q:v 2 output/frame_%06d.jpg. The -q:v 2 flag produces high-quality JPEG at roughly 200-400 KB per frame for 4K, which is 10x more storage-efficient than PNG while maintaining sufficient quality for vision model inference. For depth estimation and segmentation that require lossless input, extract a subsample at lower rate (e.g., 5 FPS) as 16-bit PNG. Store extracted frames in a separate S3 prefix: s3://your-bucket/frames/{clip_id}/rgb/frame_{NNNNNN}.jpg.

Run immediate quality checks during ingestion. Verify the video file is not corrupted using ffprobe (check that duration, codec, and frame count are parseable). Compute per-frame blur scores using the Laplacian variance method (cv2.Laplacian(gray, cv2.CV_64F).var()) on every 30th frame, flagging clips where more than 30% of sampled frames have blur scores below 100. Compute exposure histograms and flag clips with more than 20% over-exposed or under-exposed frames. These automated checks catch the most common collection quality issues (lens smudges, camera tilted to ceiling, recording started before camera was properly worn) before any expensive GPU processing runs.

Egocentric video inherently captures bystander faces, readable text, and other sensitive content. Build a robust de-identification pipeline that runs before any enrichment. Use RetinaFace (the mobilenet0.25 backbone for speed, or ResNet-50 for higher recall) to detect faces at 1 FPS. For each detected face bounding box, expand the box by 20% in each direction to ensure hair and ears are covered, then apply a strong Gaussian blur (kernel size 99, sigma 30) to the expanded region. Track face detections across frames using a simple IOU tracker to ensure consistent blurring even if the detector misses occasional frames.

Add a text/screen detection layer. Run PaddleOCR's detection model (not recognition, just detection) on every 5th frame to find regions containing readable text. Blur detected text regions with the same Gaussian kernel. This catches computer screens, phone screens, document text, and name badges that appear in egocentric footage.

For license plates, use a specialized detector (WPOD-NET or a YOLO model fine-tuned on plate detection) if your collection includes any outdoor or parking lot scenarios. Render the blurred video using FFmpeg's drawbox filter applied programmatically, producing a new video file at the same resolution and framerate as the original. Store both the original (in an access-restricted bucket with encryption at rest) and the blurred version. Generate a PII detection report for each clip that lists: number of faces detected, number of text regions detected, number of plates detected, and frames where detection confidence was low (these are the highest-risk frames for missed PII). Human reviewers should audit clips with high PII density (more than 50 face detections per minute) or low-confidence detections, watching at 2x speed and flagging any unblurred PII.

Process each privacy-filtered clip through a stack of vision models that extract the training signals downstream models need. Run these in a defined order with GPU batching for efficiency.

First, monocular depth estimation. Load the Depth Anything V2 ViT-Large checkpoint and process frames at the model's native resolution (518x518). Batch 8-16 frames per GPU forward pass depending on available VRAM (40 GB A100 fits batch size 16). Save depth maps as 16-bit PNG (values in millimeters) alongside the original frames. On an A100, expect 15-20 FPS throughput, meaning a 5-minute clip at 30 FPS (9,000 frames) takes approximately 8-10 minutes.

Second, semantic segmentation using SAM2 (Segment Anything Model 2) in video mode. Initialize SAM2 with automatic mask generation on the first frame, then propagate masks through the video using SAM2's memory-based tracking. This produces instance-level masks with temporal consistency across frames. Post-process masks to assign semantic categories by running CLIP on cropped mask regions and matching against a category vocabulary (hand, tool, food, container, appliance, furniture, person, background). Save masks as per-frame indexed PNG where each pixel value maps to an instance ID, with a sidecar JSON mapping instance IDs to semantic categories.

Third, hand-object interaction detection. Use a model trained on the Ego4D hand-object interaction benchmark (such as the 100DOH detector) to extract per-frame annotations of hand state (free, pre-grasp, grasp, post-release) and contacted object bounding boxes. Save as JSON per frame with fields: left_hand_state, right_hand_state, left_contact_object_bbox, right_contact_object_bbox.

Fourth, dense captioning. Sample frames at 2-second intervals and send to Gemini 2.5 Flash or GPT-4o with a prompt: 'Describe what the person wearing the camera is doing in this frame. Include objects being manipulated, the action being performed, and the environment. Be specific and concise (15-30 words).' Save captions with their timestamps. Compute CLIP similarity between each caption and its source frame as an automated quality check.

Fifth, generate per-clip vector embeddings by encoding 5 uniformly sampled frames through CLIP ViT-L/14 and averaging the embeddings into a single 768-dimensional vector. Store these in the metadata database for similarity search and dataset curation.

After enrichment, run a comprehensive validation pass that catches processing errors and assesses data quality across the entire collection. Implement automated validators for each enrichment output. For depth maps: verify that no depth map is all-zero (processing failure), compute the depth range per frame and flag maps where the maximum depth is below 0.5 meters or above 20 meters (likely sensor or model error), and check temporal consistency by computing the mean absolute depth change between consecutive frames (should be below 0.3 meters for stationary scenes). For segmentation masks: verify that hand masks appear in at least 40% of frames for manipulation-focused data (catching clips where the camera was angled away from hands), check that no single instance ID covers more than 80% of pixels (catching degenerate all-background masks), and verify mask temporal consistency using IOU between consecutive frames.

Build a quality dashboard that displays aggregate statistics across the pipeline. Track: total clips processed and in-progress, processing time per stage (ingestion, depth, segmentation, captioning), failure rate per stage with error categorization, per-clip quality scores (composite of blur, exposure, depth validity, mask coverage, and caption relevance), and collection coverage metrics (hours per activity category, hours per environment type, hours per collector). Use this dashboard for active curation: sort clips by quality score and review the bottom 10% to decide whether to re-process, re-collect, or discard.

For datasets where diversity matters, build a coverage matrix that cross-tabulates activity categories against environment types and identifies under-represented cells. Feed this back to the collection team as targeted collection assignments: 'We need 10 more hours of cooking activities in non-kitchen environments' or '5 more hours of workshop activities with power tools.' This feedback loop between pipeline quality metrics and collection planning is what distinguishes a production pipeline from a one-shot processing job.

Convert the pipeline outputs into the format your downstream consumers expect. For embodied AI research teams training on egocentric data, the standard delivery format is a directory structure per clip: /{clip_id}/rgb/ (JPEG frames), /{clip_id}/depth/ (16-bit PNG depth maps), /{clip_id}/segmentation/ (indexed PNG masks plus category JSON), /{clip_id}/hand_object/ (per-frame interaction JSON), /{clip_id}/captions.json (timestamped captions), and /{clip_id}/metadata.json (clip-level metadata including quality scores, camera parameters, and collection context). Compress each clip directory into a .tar.gz archive for efficient transfer.

For teams using WebDataset (the standard for large-scale PyTorch training), convert each clip into a sequence of .tar shards where each sample is a single timestep containing rgb.jpg, depth.png, mask.png, caption.txt, and metadata.json. This format enables streaming DataLoader access without downloading the full dataset. For HuggingFace Datasets integration, generate a dataset loading script that uses datasets.GeneratorBasedBuilder to yield samples from the tar archives.

Create a dataset manifest that serves as the entry point for the delivery. The manifest should be a Parquet file with one row per clip containing: clip_id, duration, activity_category, environment_type, quality_score, frame_count, enrichment_completeness (bitfield indicating which enrichment layers are present), and S3 paths to all artifacts. Include a Python SDK (a single .py file with 200-300 lines) that wraps the manifest and provides methods like dataset.clips(activity='cooking', min_quality=0.8) for filtering, dataset.load_clip(clip_id) for loading all modalities, and dataset.stream() for WebDataset-style streaming. This SDK eliminates the 1-2 weeks that teams typically spend writing custom data loading code for new datasets.