Language-Conditioned Policy: Natural Language Control for Robot Manipulation

A language-conditioned policy is a robot control model that takes both sensory observations (images, depth maps, proprioception) and a natural language instruction as input, and outputs motor actions that execute the described task. The language instruction serves as a task specification, enabling a single policy to perform many different tasks depending on what it is told to do, rather than requiring a separate policy per task.

Formally, a language-conditioned policy is a function pi(a | o, l) that maps an observation o and language instruction l to an action distribution over a. The observation is typically one or more camera images, possibly augmented with robot proprioception (joint angles, gripper state). The language is processed by a frozen or fine-tuned language encoder that produces an embedding vector or token sequence. The action space is usually continuous (joint velocities, end-effector deltas) or discretized into bins. The policy is trained on demonstration data consisting of (observation, action, language) tuples collected through teleoperation or kinesthetic teaching.

The architecture of language-conditioned policies has evolved rapidly. Early approaches like CLIPort (Shridhar et al., 2022) used CLIP to encode language and images into a shared embedding space, predicting spatial action maps for pick-and-place. RT-1 (Brohan et al., 2023) trained a transformer policy on 130,000 demonstrations spanning 700+ tasks, achieving 97% success on seen tasks. RT-2 (Brohan et al., 2023) fine-tuned a vision-language model (PaLM-E) to output discretized actions, demonstrating that internet-scale language pretraining transfers to robot control. OpenVLA (Kim et al., 2024) showed that open-source VLAs fine-tuned from Llama-based vision-language models achieve competitive performance with a fraction of RT-2's compute.

The critical insight driving this architecture is that language provides a natural interface for compositional task specification. Instead of collecting separate demonstration datasets for 'pick up the red cup', 'pick up the blue plate', and 'pick up the green bowl', a language-conditioned policy learns the compositional structure — mapping color adjectives to visual features and object nouns to shape categories — enabling generalization to novel color-object combinations never seen in training.

Building effective language-conditioned policies requires careful attention to three aspects of the training data: language diversity, demonstration coverage, and language-action alignment.

Language diversity determines whether the policy learns genuine language understanding or superficial template matching. If training instructions follow a rigid pattern ('pick up the [color] [object]'), the policy may fail when users give natural instructions ('grab that thing over there'). Best practice is to collect 5-10 distinct language paraphrases per task from multiple annotators, including formal imperative commands ('place the cup on the saucer'), informal requests ('could you put that cup down?'), goal-state descriptions ('the cup should be on the saucer'), and spatial references ('move it to the left of the plate'). Automated paraphrase diversity scoring ensures no two descriptions for the same task are too similar.

Demonstration coverage must span the compositional space of objects, actions, and spatial relations that the policy will encounter. A policy trained only on 'pick up' tasks cannot generalize to 'pour' or 'push' actions regardless of language understanding. The demonstration set should cover the cross-product of target objects, action primitives, and spatial configurations, with enough examples per combination (10-50 demonstrations) for the policy to learn the sensorimotor mapping. Coverage gaps — object-action combinations missing from training — are the primary failure mode in deployment.

Language-action alignment quality determines whether the policy learns the correct mapping between words and behaviors. Misaligned labels — an instruction saying 'pick up the cup' paired with a demonstration of placing the cup — create contradictory training signal. Alignment is best ensured by collecting language annotations contemporaneously with demonstrations (operators narrate as they teleop) rather than post-hoc (annotators describe previously recorded clips). When post-hoc annotation is necessary, Claru implements a verification step where a second annotator watches the demonstration and confirms the language description is accurate, flagging misalignments for correction.

Claru produces the language-paired demonstration data that language-conditioned policies and VLA models require. Our teleoperation campaigns capture manipulation demonstrations with real-time operator narration, ensuring tight temporal alignment between language and action. Our annotation pipeline then generates 5-10 diverse paraphrases per task from multiple annotators, covering the range of natural language expressions that deployed systems will encounter. Language quality controls enforce paraphrase diversity (no two descriptions share more than 40% of content words), grounding accuracy (each instruction unambiguously matches the demonstrated behavior), and compositional coverage (object-action-spatial combinations are systematically represented). With 386,000+ annotated clips and growing, Claru provides the scale and language variety that general-purpose language-conditioned policies demand.