Data Enrichment: Definition, Applications, and Training Data Requirements

Data enrichment is the process of adding structured information to raw data samples to make them usable for machine learning training and evaluation. In its simplest form, enrichment attaches metadata — timestamps, source identifiers, capture conditions — that enable dataset management and filtering. In its most sophisticated form, enrichment layers multiple derived annotations onto each sample: natural language descriptions, depth maps, segmentation masks, object detection labels, activity classifications, quality scores, and embedding vectors for similarity search.

For physical AI and robotics datasets, enrichment is the bridge between raw sensor captures and training-ready data. A raw egocentric video clip contains only pixel values and audio. After enrichment, that same clip carries: a natural language caption describing the activity ("person opens kitchen drawer and retrieves a spatula"), temporal activity boundaries, hand and object bounding boxes, a monocular depth map, a scene complexity score, visual quality metrics (blur, exposure, noise), CLIP/DINOv2 embedding vectors, and structured metadata about the recording environment. Each of these enrichment layers serves specific downstream consumers — captions enable language-conditioned policy training, depth maps enable spatial reasoning, embeddings enable similarity search and deduplication.

The enrichment pipeline architecture typically follows a directed acyclic graph (DAG) where each enrichment stage reads the original data plus outputs from previous stages. Raw video enters the pipeline and flows through: (1) technical metadata extraction (resolution, codec, duration, frame rate), (2) visual quality scoring (blur detection, exposure analysis, noise estimation), (3) scene understanding (object detection, depth estimation, semantic segmentation), (4) activity recognition (temporal action boundaries, verb-noun labels), (5) natural language captioning (dense descriptions from vision-language models), and (6) embedding generation (vector representations for search and clustering). Each stage writes its output to a structured schema alongside the original data, building a progressively richer representation.

The design tension in enrichment is between automation and accuracy. Fully automated pipelines using foundation models (CLIP, SAM, Depth Anything, Gemini, GPT-4V) process thousands of samples per hour at negligible marginal cost but produce noisy annotations. Fully human pipelines produce high-accuracy annotations but at 100-1000x the cost and 10-100x slower throughput. Production enrichment systems operate in the space between: automated models generate draft annotations, quality scoring identifies samples where model confidence is low, and human annotators review and correct only the flagged subset.

Building a production enrichment pipeline requires decisions across four dimensions: what to enrich, which models to use, how to validate quality, and how to store and serve the enriched data.

The enrichment scope should be driven by downstream model requirements. If you are training a VLA model, you need language descriptions, action labels, and camera intrinsics — not necessarily depth maps or segmentation masks. If you are training a world model, you need dense scene representations including depth, optical flow, and object instance tracking. Enriching data with annotations that no downstream consumer uses wastes compute and storage.

Model selection for automated enrichment should prioritize the best available model for each modality rather than using a single multimodal model for everything. For text captioning, Gemini 2.0 Flash or GPT-4o provide the best detail-to-cost ratio. For depth estimation, Depth Anything V2 with the ViT-L backbone produces the most consistent monocular depth maps. For object detection, Grounding DINO handles open-vocabulary detection better than fixed-category detectors. For segmentation, SAM 2 provides the best zero-shot instance masks. Running dedicated models per modality costs more compute but produces significantly better annotations than routing everything through a single VLM.

Quality validation must be built into the pipeline, not bolted on afterward. Each enrichment stage should output a confidence score alongside its annotation. Samples where any enrichment stage has confidence below a threshold are flagged for human review. Periodic audits on random samples verify that automated annotations meet accuracy targets. For robotics data, the critical quality check is cross-modal consistency: the language caption should describe the same activity shown in the video, the depth map should be geometrically consistent with the RGB frames, and the object labels should match the objects visible in the scene.

Claru's enrichment pipeline processes over 1 million clips with multi-provider enrichment: 3-4 vision-language models generate captions and scene descriptions, Depth Anything V2 produces depth maps, and custom quality models score visual and annotation quality. The pipeline writes all enrichment outputs to a structured database with versioned schemas, enabling clients to query specific enrichment layers and filter by quality thresholds for their particular training needs.

Data enrichment is the core of Claru's data pipeline. Every clip in our catalog of 386,000+ annotated egocentric videos passes through a multi-stage enrichment pipeline that produces natural language captions from multiple vision-language models, monocular depth maps from Depth Anything V2, visual quality scores, scene complexity metrics, and 768-dimensional embedding vectors for similarity search and deduplication.

For custom data projects, Claru designs enrichment pipelines tailored to each client's model architecture. Teams training VLA models receive language-enriched demonstrations with action-aligned captions. Teams training world models receive dense scene representations with depth, optical flow, and object state annotations. Teams building perception systems receive detection and segmentation enrichments calibrated to their specific object categories. Every enrichment layer includes confidence scores and quality metrics, and clients can request human verification of any enrichment stage where automated model accuracy falls below their quality threshold.