How to Build a Language-Conditioned Robot Dataset

Before collecting a single demonstration, formalize the relationship between language and robot behavior in your domain. Create a task taxonomy that maps every target behavior to a canonical instruction template, then expand each template with paraphrases. For example, the canonical template 'pick up {object} from {location}' might have paraphrases like 'grab the {object} off the {location}', 'take the {object} sitting on the {location}', and 'get the {object} from the {location}'. Store this ontology in a structured YAML or JSON file that becomes the ground truth for annotator training.

Define the language specification across three levels of granularity. Goal-level instructions describe the desired end state ('put all fruits in the bowl'). Step-level instructions describe individual manipulation primitives ('reach for the apple on the left'). Motion-level narrations describe continuous trajectories ('move the arm slowly to the right'). Specify which levels your target model requires. RT-2-style VLAs typically need goal-level only, while hierarchical planners like SayCan require all three levels. Document the expected vocabulary size (aim for 200+ unique templates), the minimum paraphrases per template (3-5), and the spatial/attribute vocabulary (colors, sizes, relative positions, container types). This specification document prevents annotation drift and ensures consistency when you scale from a pilot of 50 episodes to a production run of 10,000+.

Build a recording system that captures synchronized robot observations, actions, and language in a single pipeline. The core stack should include ROS2 for sensor coordination, with a custom rosbag2 recorder that writes RGB frames (at least 640x480 at 30 Hz from both a wrist-mounted Intel RealSense D405 and an external D435), joint states at the robot's native control rate (typically 100-500 Hz for arms like Franka Emika or UR5e), and gripper state. Add a dedicated language channel: either a microphone input for real-time narration (use a lapel mic with 16 kHz sampling, saved as WAV alongside the rosbag) or a text input terminal where a co-located annotator types instructions timestamped to the robot clock.

Critical pitfall: clock synchronization. Use PTP (Precision Time Protocol) or at minimum NTP to synchronize the robot controller, camera, and annotation workstation clocks to within 1 ms. Without this, language timestamps will drift relative to actions, making temporal grounding unreliable. Test synchronization by recording a calibration sequence where you trigger a visible event (a light flash) while simultaneously logging a text annotation, then verify the timestamps align within your tolerance. Store all data in HDF5 with a hierarchical layout: /episode_N/observations/rgb_wrist, /episode_N/observations/rgb_external, /episode_N/actions/joint_positions, /episode_N/language/instructions. Include metadata fields for episode ID, operator ID, task template ID, and environment configuration hash.

Run data collection in two phases. Phase one is a scaffolded pilot (50-100 episodes) where the teleoperator and a language annotator work side by side. The annotator speaks or types the goal instruction before the episode begins, narrates sub-steps during execution, and confirms task completion at the end. Use this pilot to calibrate the language ontology: identify instructions that are confusing, tasks that need new templates, and spatial references that are ambiguous. Refine the ontology YAML after the pilot and freeze it before scaling.

Phase two is production collection. For throughput, switch to a two-pass approach: the operator performs demonstrations silently at full speed (targeting 20-40 episodes per hour depending on task complexity), and a separate team adds language post-hoc from video replay. Use Label Studio with a custom video annotation interface that lets annotators scrub through the episode, place temporal markers at sub-task boundaries, and type instructions aligned to each segment. Configure Label Studio's interface template to enforce minimum instruction length (6 words) and require selection of the matching task template from a dropdown. This catches terse or off-ontology annotations immediately. Track collection progress using a dashboard that shows episodes per task variant, language diversity metrics (unique instruction count, type-token ratio), and annotator throughput. Flag any task variant with fewer than 50 episodes or fewer than 10 unique instruction phrasings for additional collection.

Raw human annotations alone rarely achieve sufficient language diversity for robust generalization. Augment the dataset systematically using both template-based and model-based paraphrasing. First, apply your ontology templates: for each episode's human-written instruction, generate all applicable template variants by substituting synonyms from a controlled vocabulary (e.g., 'pick up' to 'grab', 'take', 'lift'; 'red mug' to 'crimson cup', 'red coffee mug'). Store generated paraphrases in a separate annotation field so you can distinguish human-written from synthetic instructions during analysis.

Second, use a language model to generate naturalistic paraphrases. Feed each instruction to an instruction-tuned model (Llama 3 70B or GPT-4) with a prompt like 'Rephrase this robot instruction five different ways, preserving the exact meaning and all spatial references: [instruction]'. Filter generated paraphrases using CLIP text embeddings: compute cosine similarity between the original and each paraphrase, keeping only those with similarity above 0.85, which preserves semantic content while allowing lexical variation. Discard any paraphrase that introduces spatial references not present in the original. Run a human verification pass on a 10% sample of generated paraphrases to measure hallucination rate, targeting below 5%. Finally, compute a SentenceTransformers embedding (using all-MiniLM-L6-v2) for every instruction in the dataset and visualize the embedding space with UMAP. Look for clusters that are too tight (low diversity) or isolated points (potential annotation errors).

This step ensures that language instructions actually correspond to the demonstrated robot behavior, which is the single most important quality signal in a language-conditioned dataset. Run three validation passes. First, automated alignment scoring: for each episode, extract the CLIP embedding of the goal instruction and the CLIP embedding of the final frame, then compute their cosine similarity. Episodes where the language-vision similarity falls below 0.3 are likely misaligned. Extract these for human review. Second, temporal alignment check: for datasets with sub-step annotations, verify that each instruction's timestamp falls within the corresponding action segment. Compute the mean absolute temporal error across the dataset and flag any episode where an instruction timestamp is more than 1 second outside its action segment boundaries.

Third, human validation: sample 200-500 episodes stratified by task type and annotator. Show each episode's video alongside its language annotations to three independent reviewers who rate alignment on a 1-5 scale. Compute Krippendorff's alpha to measure inter-rater reliability, targeting alpha above 0.75. Any episode rated below 3 by two or more reviewers goes back for re-annotation. Aggregate the results to identify systematic issues: if a particular annotator consistently produces low-alignment annotations, retrain or remove them. If a specific task type has uniformly low alignment scores, the task definition or language template likely needs revision. This validation pass typically catches 5-15% of episodes that need correction. The cost of this step is significant (expect 2-3 hours of reviewer time per 1,000 episodes) but skipping it reliably degrades downstream policy performance by 10-20% on held-out instruction following benchmarks.

Convert the validated dataset into the format your target model expects. For RT-2-style models, this means RLDS (the TensorFlow Datasets format used by Open X-Embodiment) where each episode is a tf.data.Dataset with features for image observations, actions, language instructions, and metadata. Use the rlds_dataset_builder template from Google's rlds repository to scaffold your builder class. Specify features precisely: images as tf.uint8 tensors of shape (H, W, 3), actions as tf.float32 of shape (action_dim,), and language as tf.string. For Octo or OpenVLA, the same RLDS format works but ensure your action normalization matches their conventions (typically zero-mean unit-variance computed per action dimension across the training split).

Generate train/validation/test splits with care. Split by environment configuration or scene, not randomly by episode, to avoid data leakage where the model memorizes specific table layouts. Reserve at least 10% of unique task-object combinations for the test set to evaluate compositional generalization (can the model follow 'pick up the green cup' if it only saw 'pick up the blue cup' and 'pick up the green bottle' during training). Publish a dataset card following the Datasheets for Datasets template that documents: total episodes, unique instructions, vocabulary size, language diversity metrics, annotator count, collection time span, robot platform, camera specifications, known biases (e.g., right-handed operator bias), and intended use cases. Include a Python loading script that returns a PyTorch DataLoader or tf.data.Dataset pipeline, along with a visualization notebook that renders sample episodes with overlaid language annotations. This documentation is not optional: teams that receive undocumented datasets spend 1-2 weeks just understanding the data format before they can begin training.