Egocentric Video Data Collection for Robotics and World Modeling
Challenge:Robotic manipulation and world-model research requires massive volumes of first-person video showing natural human behavior in diverse real-world environments, and no public dataset meets frontier labs' quality or diversity bar.
Solution:We built a purpose-built capture and ingestion platform — not adapted from an off-the-shelf tool — and launched three parallel pipelines within days of engagement, each optimized for different environments and interaction types.
Result:The dataset became primary training data for the lab's world-modeling and robotic manipulation research.
Robotic manipulation and world-model research requires massive volumes of first-person video showing natural human behavior in diverse real-world environments, and no public dataset meets frontier labs' quality or diversity bar. The lab had attempted internal collection but stalled at modest scale due to hardware logistics, contributor recruitment across geographies, and the overhead of enforcing consistent capture quality — resolution, frame rate, scene coverage, and activity diversity — across hundreds of participants. They needed a partner who could launch a production-ready pipeline in days, not months, and adapt to evolving research specifications on a weekly cadence without sacrificing annotation-ready quality standards.
We built a purpose-built capture and ingestion platform — not adapted from an off-the-shelf tool — and launched three parallel pipelines within days of engagement, each optimized for different environments and interaction types.
The first pipeline deployed GoPro and DJI wearable cameras for high-fidelity, wide-angle egocentric capture of manipulation tasks, cooking, and locomotion — producing 219,000+ clips. The second pipeline used smartphone cameras for rapid, high-volume capture of everyday activities across diverse indoor and outdoor environments — producing 155,000+ clips. The third pipeline targeted specific activity categories (pouring, cutting, assembling, folding, fastening) with structured task instructions — producing 12,000+ precisely labeled clips.
A structured activity taxonomy was co-developed with the lab's research team through three iterative revision cycles — testing draft categories against real captured footage to resolve gaps and ambiguities. The final taxonomy organized activities by environment (kitchen, workshop, outdoor), motor complexity (gross motor, fine manipulation, locomotion), and interaction type (tool use, object transfer, environmental navigation). Every clip was annotated with structured activity classifications and enriched metadata — environment type, lighting conditions, number of objects involved, and interaction complexity score — using a labeling interface that enforced taxonomy compliance at the UI level.
Automated checks at upload time validated resolution, duration, orientation, and file integrity before clips entered the annotation pipeline. An in-house QA team then ran continuous validation on every submission within 24 hours. Real-time dashboards enabled dynamic rebalancing across geographies and activity types as the lab's priorities shifted. The operational model was built for velocity: contributor onboarding took under 48 hours, QA turnaround was same-day, and weekly delivery batches kept the lab's training pipeline fed continuously.
First-person footage on Explore
Public first-person clips from Claru Explore across home, street, retail and tabletop settings, shown muted.
- Upload-time checksResolution, duration, orientation and file integrity validated before a clip enters annotation.
- 24-hour human reviewAn in-house QA team reviewed every submission within 24 hours.
- Taxonomy enforced in the UIThe labeling interface accepts only taxonomy labels, so free-text drift cannot creep in.
- Live rebalancingDashboards tracked pass rates by geography and activity type, so quality dips were caught per segment.
Clips by pipeline
- Wearable capture (GoPro, DJI)
- 219K
- Smartphone capture
- 155K
- Activity-specific capture
- 12K+
Bar length is proportional to clips. The three streams total 386K+ clips.
Annotated first-person clips, delivered in weekly batches.
- Clips from three pipelines
- Wearable, smartphone and activity-specific streams, normalized to consistent resolution and frame-rate specifications.
- Activity classification
- Structured labels from the co-developed taxonomy: environment, motor complexity and interaction type.
- Enriched metadata
- Environment type, lighting conditions, number of objects involved and an interaction complexity score per clip.
- Weekly batches
- Delivered weekly, so training runs could start during collection.
The dataset became primary training data for the lab's world-modeling and robotic manipulation research. The three-pipeline architecture meant each stream could be independently tuned — GoPro footage for high-fidelity manipulation, smartphone for environmental diversity, activity-specific for rare interaction types. Geographic and demographic diversity across approximately 500 contributors reduced the distribution shift between training data and real-world deployment. Weekly delivery cadence allowed the lab to begin training runs during collection rather than after, compressing their research iteration cycle by weeks.
For robotics and world models, progress depends on how quickly real-world data can be collected and put to use. Foundation models for embodied AI need millions of diverse egocentric frames showing natural human behavior across varied environments. Collecting this data requires coordinating hundreds of people across geographies, managing heterogeneous capture hardware, and enforcing research-grade quality at scale — problems that are operational, not algorithmic. This engagement produced 386K+ clips across three parallel pipelines, demonstrating that large-scale first-person video collection can be both fast and cost-effective without sacrificing the quality that frontier labs require.
Representative record from the annotation pipeline.
- How to Collect Egocentric Video DataCapture specs, devices and contributor setup
- How to Build an Egocentric Video Data PipelineFrom upload checks to delivery
- Egocentric videoWhat first-person data is and how it is used
- Egocentric Outdoor Urban Video DatasetSidewalks, crosswalks and plazas
- Explore: Sidewalks & pedestrian waysPublic first-person walking clips
Ready to build your next dataset?
Tell us about your project and we will scope a plan within 48 hours.