Training Data for VC-1

Claru provides both the egocentric video for VC-1-style visual pretraining and the robot demonstrations for downstream policy training on VC-1 features. Our 3M+ human activity video clips span domains beyond Ego4D's coverage -- workplace operations, warehouse tasks, retail environments, and domestic activities across 100+ cities -- providing the environmental diversity that the VC-1 paper identified as critical for robust features. For downstream policy training, we deliver 224x224 RGB frames with action labels and can provide pre-extracted VC-1 features (1024-dim vectors) alongside raw images to accelerate training. For teams extending VC-1, we curate domain-specific video corpora filtered for hand-object interactions, target activity types, or specific environments, matching the paper's finding that data curation matters more than indiscriminate scaling.

VC-1 (Visual Cortex 1) is a visual foundation model for embodied AI developed by Meta AI, published by Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, and colleagues in 2023 (arXiv 2303.18240, presented at NeurIPS 2023). The paper presents the largest and most comprehensive empirical study of pretrained visual representations (PVRs) for robotics, and VC-1 is the model that emerged from this study -- a Vision Transformer (ViT-L, 307M parameters) pretrained via Masked Auto-Encoding (MAE) on a carefully curated mixture of egocentric video and web images that outperforms all prior visual representations on the authors' CortexBench evaluation suite.

Unlike end-to-end VLA models (RT-1, OpenVLA) that map directly from pixels to actions, VC-1 is a visual encoder -- it produces rich feature representations from images that are then consumed by downstream policy heads for specific tasks. This separation of visual representation learning from policy learning is a fundamentally different design philosophy. The thesis is that a general-purpose visual understanding model, pretrained on large-scale human activity data, provides better features for robot learning than task-specific representations or representations trained on generic web images (like ImageNet).

The paper's core contribution is methodological rigor. Rather than proposing a single model, the authors systematically varied pretraining data composition (which datasets, what mixture), model size (ViT-S through ViT-L), and pretraining objective (MAE, contrastive, supervised) across 17 downstream tasks spanning locomotion, navigation, dexterous manipulation, and mobile manipulation. This produced a comprehensive map of what works and what doesn't in visual pretraining for embodied AI. The winning configuration -- ViT-L with MAE pretraining on egocentric video + ImageNet -- became VC-1.

VC-1's practical significance is that it provides an open-source, high-quality visual backbone that any robot learning team can use as a drop-in replacement for ImageNet-pretrained features. The pretrained weights, training code, and evaluation suite are all publicly available, and the model has been adopted across multiple research groups as a standard visual encoder for robot policies.

VC-1's architecture is a standard Vision Transformer Large (ViT-L/16) with 307M parameters, 24 transformer layers, 1024-dim hidden representations, and 16x16 pixel patches. The model is pretrained using Masked Auto-Encoding (MAE): 75% of image patches are randomly masked, and the model learns to reconstruct the missing pixels from the visible patches. This self-supervised objective forces the model to develop rich spatial understanding without requiring any labels.

The paper's key innovation is not the architecture but the systematic study of pretraining data composition. The authors created datasets with different mixtures of: (1) Ego4D -- 3,670 hours of egocentric video from daily activities. (2) Six additional egocentric video sources (Epic-Kitchens, Something-Something-v2, 100 Days of Hands, and others) totaling 4,000+ hours across 7 datasets. (3) ImageNet -- 1.2M static web images. (4) Robot manipulation datasets. They trained ViT models at multiple scales on different slices of this data to understand which combinations produce the best downstream features.

The critical finding was that data mixture matters enormously, and the best mixture depends on the downstream task domain. For manipulation tasks, egocentric video (particularly hand-object interaction footage) provides the strongest features. For navigation, ImageNet contributes more. The overall best configuration -- VC-1 -- uses a mixture of all egocentric video sources plus ImageNet, trained with MAE at ViT-L scale. This mixture outperforms any single data source, and the egocentric video component is essential: removing it degrades manipulation task features by 10-20%.

Another surprising finding was that scaling dataset size and diversity does not universally improve performance. Adding more data helps on average, but for specific tasks, the wrong data can actually degrade performance. For example, adding navigation-heavy video data slightly hurts dexterous manipulation features. This suggests that teams building VC-1-style models should curate their pretraining data for their target task domain rather than blindly maximizing scale.

VC-1's pretraining corpus consists of two main components. The egocentric video component draws from 7 datasets totaling over 4,000 hours: Ego4D (3,670 hours of daily activities across 74 locations in 9 countries), Epic-Kitchens (100 hours of kitchen activity), Something-Something-v2 (220,847 video clips of object interactions), 100 Days of Hands, and three additional egocentric sources. From these videos, the authors sampled 4.3 million individual frames for MAE pretraining. The static image component is ImageNet (1.2 million images), which provides object-centric visual features that complement the egocentric video's activity-centric features.

For MAE pretraining, each training sample is a single 224x224 RGB image -- no temporal structure, no action labels, no language annotations. The images are resized and normalized to ImageNet statistics (mean and standard deviation per channel). The 75% masking ratio means the model only sees 49 of the 196 patches per image during training, forcing it to learn strong spatial priors. Training the ViT-L model to convergence required over 10,000 GPU-hours on NVIDIA A100 GPUs.

For downstream policy training (not VC-1 pretraining, but using VC-1 features), the data requirements depend on the specific task. The CortexBench evaluation includes tasks with as few as 25 demonstrations (dexterous manipulation in simulation) and as many as 50,000 episodes (navigation in Habitat). The VC-1 features reduce the data requirements for downstream tasks compared to training from scratch, but the exact reduction varies by task. On manipulation tasks, VC-1 features typically enable competitive performance with 2-5x less downstream data than training a CNN from scratch.

For teams building VC-1-style visual encoders, the pretraining data curation is the most important decision. The paper shows that indiscriminate data aggregation can hurt specific tasks -- you want egocentric hand-object interaction video for manipulation, navigation footage for navigation, and so on. The Ego4D dataset is the single most impactful data source because it provides massive scale (3,670 hours) with diverse manipulation activities. Adding ImageNet on top helps because it provides complementary object-centric features (isolated objects on clean backgrounds, which aids object recognition in cluttered robot scenes).

Claru provides both the egocentric video data for VC-1-style visual pretraining and the robot demonstration data for downstream policy training on VC-1 features. For pretraining, our catalog of 3M+ human activity video clips spans domains beyond Ego4D's coverage: workplace operations, warehouse activities, retail environments, outdoor tasks, and culturally diverse domestic settings collected across 100+ cities. These videos provide the hand-object interaction footage that VC-1's paper identified as the most impactful pretraining data for manipulation tasks.

A key advantage of Claru's pretraining data is environmental diversity. Ego4D, while large (3,670 hours), is concentrated in specific geographic and cultural contexts. The VC-1 paper showed that data diversity matters -- broader visual diversity in pretraining produces more robust features. Claru's global collection network naturally provides this diversity: our egocentric videos capture manipulation activities in kitchens, offices, factories, and outdoor settings across different lighting conditions, object types, and cultural contexts that expand the visual vocabulary beyond what any single academic dataset covers.

For downstream policy training, Claru delivers robot demonstration datasets with VC-1-compatible observations. Our data includes 224x224 RGB frames (matching VC-1's expected input resolution), action labels in the downstream policy's native format, and language annotations. We can deliver data pre-processed with VC-1 feature extraction -- running the frozen VC-1 encoder on our RGB frames and delivering the resulting 1024-dim feature vectors alongside the raw images -- which significantly accelerates policy training for teams that want to skip the feature extraction step during training.

We also support teams that want to extend or retrain VC-1 itself. Our video data pipeline can deliver frame-level samples at any target resolution with consistent normalization, formatted for MAE pretraining. We provide dataset metadata including activity labels, hand-object contact segments, and scene descriptions that enable targeted curation -- for example, filtering for hand-object interaction frames to create a manipulation-focused pretraining corpus, which the VC-1 paper identified as the most effective data strategy for manipulation-centric embodied AI.