Data Collection

Egocentric Video Data Collection for Robotics and World Modeling

386K+First-person video clips captured
summary.md

Challenge:Robotic manipulation and world-model research requires massive volumes of first-person video showing natural human behavior in diverse real-world environments, and no public dataset meets frontier labs' quality or diversity bar.

Solution:We built a purpose-built capture and ingestion platform — not adapted from an off-the-shelf tool — and launched three parallel pipelines within days of engagement, each optimized for different environments and interaction types.

Result:The dataset became primary training data for the lab's world-modeling and robotic manipulation research.

386K+Total first-person video clips captured
219KGoPro & DJI wearable capture clips
155KSmartphone capture clips
~500Global contributors across 3 pipelines
// THE CHALLENGE

Robotic manipulation and world-model research requires massive volumes of first-person video showing natural human behavior in diverse real-world environments, and no public dataset meets frontier labs' quality or diversity bar. The lab had attempted internal collection but stalled at modest scale due to hardware logistics, contributor recruitment across geographies, and the overhead of enforcing consistent capture quality — resolution, frame rate, scene coverage, and activity diversity — across hundreds of participants. They needed a partner who could launch a production-ready pipeline in days, not months, and adapt to evolving research specifications on a weekly cadence without sacrificing annotation-ready quality standards.

// OUR APPROACH

We built a purpose-built capture and ingestion platform — not adapted from an off-the-shelf tool — and launched three parallel pipelines within days of engagement, each optimized for different environments and interaction types.

The first pipeline deployed GoPro and DJI wearable cameras for high-fidelity, wide-angle egocentric capture of manipulation tasks, cooking, and locomotion — producing 219,000+ clips. The second pipeline used smartphone cameras for rapid, high-volume capture of everyday activities across diverse indoor and outdoor environments — producing 155,000+ clips. The third pipeline targeted specific activity categories (pouring, cutting, assembling, folding, fastening) with structured task instructions — producing 12,000+ precisely labeled clips.

A structured activity taxonomy was co-developed with the lab's research team through three iterative revision cycles — testing draft categories against real captured footage to resolve gaps and ambiguities. The final taxonomy organized activities by environment (kitchen, workshop, outdoor), motor complexity (gross motor, fine manipulation, locomotion), and interaction type (tool use, object transfer, environmental navigation). Every clip was annotated with structured activity classifications and enriched metadata — environment type, lighting conditions, number of objects involved, and interaction complexity score — using a labeling interface that enforced taxonomy compliance at the UI level.

Automated checks at upload time validated resolution, duration, orientation, and file integrity before clips entered the annotation pipeline. An in-house QA team then ran continuous validation on every submission within 24 hours. Real-time dashboards enabled dynamic rebalancing across geographies and activity types as the lab's priorities shifted. The operational model was built for velocity: contributor onboarding took under 48 hours, QA turnaround was same-day, and weekly delivery batches kept the lab's training pipeline fed continuously.

01
SpecTranslate research requirements into 3 pipeline-specific task protocols
02
ActivateOnboard ~500 contributors with GoPro, DJI, and smartphone kits
03
CaptureRun 3 parallel streams — wearable, phone, activity-specific
04
ValidateSame-day QA on framing, motion, duration, and metadata
05
DeliverWeekly batches feeding the lab's training pipeline continuously
// FOOTAGE

First-person footage on Explore

Public first-person clips from Claru Explore across home, street, retail and tabletop settings, shown muted.

Preparing a drink next to a gas stove
Walking on a rough concrete surface, downward view
Holding up a small fan in a toy store
Sorting jewelry pieces over a white surface
Browse egocentric footage on Explore
// QUALITY GATES
  1. Upload-time checksResolution, duration, orientation and file integrity validated before a clip enters annotation.
  2. 24-hour human reviewAn in-house QA team reviewed every submission within 24 hours.
  3. Taxonomy enforced in the UIThe labeling interface accepts only taxonomy labels, so free-text drift cannot creep in.
  4. Live rebalancingDashboards tracked pass rates by geography and activity type, so quality dips were caught per segment.
// RESULTS
386K+Total first-person video clips captured
219KGoPro & DJI wearable capture clips
155KSmartphone capture clips
~500Global contributors across 3 pipelines

Clips by pipeline

Wearable capture (GoPro, DJI)
219K
Smartphone capture
155K
Activity-specific capture
12K+

Bar length is proportional to clips. The three streams total 386K+ clips.

// WHAT THE CUSTOMER RECEIVED

Annotated first-person clips, delivered in weekly batches.

Clips from three pipelines
Wearable, smartphone and activity-specific streams, normalized to consistent resolution and frame-rate specifications.
Activity classification
Structured labels from the co-developed taxonomy: environment, motor complexity and interaction type.
Enriched metadata
Environment type, lighting conditions, number of objects involved and an interaction complexity score per clip.
Weekly batches
Delivered weekly, so training runs could start during collection.
// IMPACT

The dataset became primary training data for the lab's world-modeling and robotic manipulation research. The three-pipeline architecture meant each stream could be independently tuned — GoPro footage for high-fidelity manipulation, smartphone for environmental diversity, activity-specific for rare interaction types. Geographic and demographic diversity across approximately 500 contributors reduced the distribution shift between training data and real-world deployment. Weekly delivery cadence allowed the lab to begin training runs during collection rather than after, compressing their research iteration cycle by weeks.

// WHY IT MATTERS

For robotics and world models, progress depends on how quickly real-world data can be collected and put to use. Foundation models for embodied AI need millions of diverse egocentric frames showing natural human behavior across varied environments. Collecting this data requires coordinating hundreds of people across geographies, managing heterogeneous capture hardware, and enforcing research-grade quality at scale — problems that are operational, not algorithmic. This engagement produced 386K+ clips across three parallel pipelines, demonstrating that large-scale first-person video collection can be both fast and cost-effective without sacrificing the quality that frontier labs require.

// DECISIONS WE MADE
Build the platform instead of adapting onewhyNo existing tool combined heterogeneous first-person capture, enforced taxonomies, enriched metadata and upload-time validation.
Three pipelines, tuned separatelywhyWearables for high-fidelity manipulation, phones for environmental diversity, activity-specific capture for rare interactions.
Co-develop the taxonomy against real footagewhyThree revision cycles resolved gaps and ambiguous categories before scale.
Parameterize the specwhyInstructions, taxonomies and thresholds update without rebuilding the pipeline; contributors received changes within 48 hours.
// SAMPLE DATA

Representative record from the annotation pipeline.

capture_pipeline_overview.json
// VIDEO SAMPLE
// CAPTURE PIPELINE BREAKDOWN
GoPro & DJI Wearable
219,598
GoPro HERO, DJI Action — 56.8% of total
Smartphone Capture
154,973
iPhone, Android — 40.1% of total
Activity-Specific
11,772
Mixed — task-guided — 3.0% of total
Total Clips386,343
// ACTIVITY CATEGORIES
Object ManipulationCookingWalkingDrivingCleaningCrafts & AssemblyPouring & CuttingFolding & FasteningPhone-based Activities
// SAMPLE SPECS
Resolution3840x2160
Aspect Ratio16:9
Duration11.08s
Frame Rate31.579 fps
Total Frames350
Bit Depth8-bit
// FAQ

Ready to build your next dataset?

Tell us about your project and we will scope a plan within 48 hours.