How to Build a Data Enrichment Pipeline for Robot Datasets

Before building the pipeline, catalog what the raw dataset contains and what enrichments will add the most value for your downstream use case. Inventory the existing fields: video frames (resolution, fps, count), proprioceptive data (joint states, EE pose), action labels, task descriptions, and any existing annotations. Then define the target enrichment schema with specific column names, data types, and storage locations.

A standard enrichment schema for robot datasets includes: caption_text (string, 50-200 words describing the scene and action), embedding_vector (float32 array of 768 dimensions from CLIP ViT-L/14), detected_objects (JSON array of {label, confidence, bbox} dicts), depth_map (float32 HxW array in meters, if no hardware depth), quality_score (float32 0-1 composite of blur, brightness, and completeness), and semantic_cluster_id (int, assigned by k-means on embeddings). Document each field with its source model, version, prompt (for VLMs), and expected update frequency. Store this schema definition as a JSON or YAML file that the pipeline reads at runtime — this makes it trivial to add new enrichment columns later without modifying pipeline code.

Most enrichments operate on individual frames rather than full video sequences, so the first pipeline stage extracts representative keyframes. For continuous video, sample uniformly at 1-2 fps — this captures all meaningful scene transitions while reducing processing volume by 15-30x compared to the native frame rate. For episode-based robot data, sample the first frame (initial state), the last frame (final state), and every Nth frame in between where N = episode_length / target_keyframes (typically 3-5 keyframes per episode).

Build a batching layer that groups keyframes into processing batches of 100-1,000 frames. For API-based enrichment (Gemini, OpenAI), respect rate limits by implementing exponential backoff with jitter (initial wait 1s, max wait 60s, jitter +/- 20%). For local GPU inference (CLIP, Florence-2), batch by model: CLIP processes 256 images per forward pass on an A100, Florence-2 handles 32 images per batch. Implement a batch processor class with methods: extract_keyframes(episode) -> List[Frame], create_batches(frames, batch_size) -> List[Batch], and process_batch(batch, enrichment_fn) -> List[Result]. This abstraction lets you swap enrichment models without changing the pipeline orchestration. Store extracted keyframes with their source episode ID and frame index so results can be mapped back to the original dataset.

Build each enrichment as an independent function that takes a batch of frames and returns structured results. Run independent enrichments in parallel to maximize throughput. The typical branches are:

(1) Caption generation: Send each keyframe to Gemini 2.5 Flash with a structured prompt: 'Describe this robot manipulation scene in 2-3 sentences. Include: the objects visible, their spatial arrangement, what the robot gripper is doing, and the apparent task goal.' Parse the response and store as caption_text. Cost: ~$0.002-0.005 per frame. (2) Embedding computation: Run CLIP ViT-L/14 or DINOv2 ViT-g/14 on each frame to produce a 768-dim or 1536-dim float32 vector. Store as a binary blob or float array. Local GPU throughput: ~500 frames/second on A100. (3) Object detection: Run Florence-2 or Grounding-DINO with text prompts listing your target object categories. Store bounding boxes as JSON arrays. Local GPU throughput: ~50 frames/second. (4) Depth estimation: Run Depth Anything V2 (ViT-L) to produce per-pixel depth maps. Store as compressed float16 arrays. Local GPU throughput: ~30 frames/second. (5) Quality scoring: Compute Laplacian variance (blur detection, threshold < 100 = blurry), mean brightness (flag < 30 or > 225), and frame completeness (no all-black or all-white regions). Combine into a 0-1 composite score.

Use Python's concurrent.futures.ThreadPoolExecutor for API calls (I/O-bound) and multiprocessing for local GPU inference (compute-bound). Write results to a staging area (temporary files or database staging table) before merging into the master dataset.

After all enrichment branches complete, merge results back into the master dataset. For HDF5 datasets, add new datasets under an /enrichments group: /enrichments/captions as variable-length strings, /enrichments/embeddings as float32 arrays, /enrichments/detections as JSON strings. For database-backed datasets, UPDATE each row with the new column values using batch upserts (1,000 rows per transaction to avoid lock contention).

Compute derived features that combine multiple enrichments: (1) Semantic clusters: run k-means (k=20-50) on the embedding vectors using scikit-learn's MiniBatchKMeans for memory efficiency. Assign each frame a cluster_id and compute cluster centroids. These clusters reveal the natural structure of your dataset (e.g., cluster 3 = grasping scenes, cluster 7 = placing scenes). (2) Deduplication flags: compute pairwise cosine similarity between embeddings within each episode and across episodes. Flag pairs with similarity > 0.95 as near-duplicates. For a 100K-frame dataset, use approximate nearest neighbors (FAISS with IVF index) to avoid the O(n^2) brute-force comparison. (3) Diversity scores: for each episode, compute the average pairwise distance between its frame embeddings — low diversity indicates repetitive content. (4) Caption-based tags: extract structured tags from captions using regex or a lightweight NLP model (e.g., spaCy entity extraction) to populate object_tags and action_tags arrays for filtering and search.

Run systematic validation on the enriched dataset. Sample 200 frames stratified across semantic clusters and manually score each enrichment: caption accuracy (1-5 scale, target mean > 4.0), object detection recall (are all visible objects detected?), depth map plausibility (do estimated depths match visible scene geometry?), and quality score calibration (do low-scored frames actually look bad?).

Build a feedback loop that uses validation results to improve the pipeline: (1) If caption accuracy is below 4.0, iterate on the VLM prompt — add few-shot examples of correct captions for your domain, specify the robot name and gripper type, and explicitly list common objects. (2) If object detection recall is low for specific categories, add those categories to the text prompt or fine-tune the detector on 50-100 manually labeled examples. (3) If quality scores do not correlate with human judgment, adjust the component weights in the composite score formula. (4) If semantic clusters are not meaningful (e.g., one cluster contains 60% of all frames), increase k or switch to HDBSCAN for variable-density clustering. Track validation metrics across pipeline versions and reject any version that regresses on previously passing metrics. Store the validation report as a JSON file alongside the dataset.

Convert the pipeline from batch processing to incremental mode for production use. Implement a checkpoint system: after each enrichment step processes a batch, record the maximum processed ID or timestamp. On the next run, query for records with IDs greater than the checkpoint. For database-backed datasets, a simple SQL query handles this: SELECT id, frame_data FROM clips WHERE ai_caption IS NULL ORDER BY id LIMIT 1000.

Deploy the pipeline as a scheduled job (cron, Airflow DAG, or Trigger.dev task) that runs every 1-6 hours depending on data arrival rate. Monitor pipeline health with metrics: frames processed per run, enrichment latency per model, API error rates, cost per frame, and quality score distribution of newly enriched frames. Alert if: error rate exceeds 5%, average quality score drops below the historical mean by more than 1 standard deviation, or a run processes zero frames when new data is expected. For cost tracking, log every API call with its token count and model name, then aggregate weekly to detect cost drift. At scale (1M+ frames), the enrichment pipeline becomes the most expensive component of the data infrastructure — monitor it accordingly.

Document the full pipeline: architecture diagram, enrichment schema, model versions and prompts, cost model, validation methodology, and runbook for common failure modes (API rate limits, GPU OOM, schema migration). This documentation is essential for handoff and reproducibility.