Language-Table Alternative: Production-Grade Language-Conditioned Robot Data

Language-Table made a landmark contribution to language-conditioned robot learning: it proved that a simple behavioral cloning policy trained on 442,000+ language-annotated demonstrations could follow 87,000 unique natural language instructions with 93.5% success. As part of OXE, it provides valuable language-grounding priors for generalist policies. However, the gap between pushing blocks on a table and performing production manipulation is enormous. Language-Table's 2D actions cannot represent grasping, lifting, or 3D manipulation. Its single top-down camera cannot capture the multi-view, multi-modal information that production policies require. And its limited object set does not prepare a model for the diversity of real-world objects. Claru provides the data that bridges this gap: language-conditioned demonstrations of full 6-DoF manipulation tasks, collected on your robot platform with the complete sensor stack (RGB, depth, force/torque, proprioception) and validated language annotations that ensure reliable language-action mapping. Teams that pretrain on OXE (including Language-Table) for language understanding and fine-tune on Claru data for task-specific manipulation achieve robust, language-instructable production policies.

Language-Table is a large-scale dataset and benchmark for language-conditioned robotic manipulation, published by Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence at Google (arXiv 2210.06407, IEEE Robotics and Automation Letters 2023). The project introduced a framework for training interactive, real-time, natural language-instructable robots and demonstrated that behavioral cloning on a sufficiently large dataset of language-annotated trajectories could produce policies that follow open-vocabulary natural language instructions in real time.

The dataset contains 442,226 human-collected demonstrations of tabletop manipulation tasks on a flat surface with colored blocks and a small number of other objects. A human teleoperator controls a robot arm (xArm with a spatula-like end-effector) via a 2D joystick to push objects around the table surface, while simultaneously providing natural language descriptions of the intended action. The observation space is a single top-down RGB camera, and the action space is 2D planar end-effector velocity commands. The resulting policy achieved a 93.5% success rate on a test set of 87,000 unique natural language strings -- an order of magnitude more commands than any previous language-conditioned robot system could handle.

The key innovation of Language-Table was demonstrating interactive language grounding: the human operator could give real-time language instructions (e.g., 'push the blue block to the left', 'make a smiley face out of blocks', 'move banana to the sum of two plus one') and the robot would respond in real time, continuously updating its behavior based on the latest language input. This represented a significant advance over prior work where language instructions were given once before execution rather than interactively during it. The dataset captures this interactive dynamic, with language annotations temporally aligned to the corresponding manipulation actions.

Language-Table is released under an Apache 2.0 license and is included in the Open X-Embodiment (OXE) collection in the RLDS format on TensorFlow Datasets. The dataset, environment (a simulated version of the tabletop setup), benchmark suite, and trained policies are all open-sourced on GitHub. While Language-Table established the viability of large-scale language-conditioned imitation learning, its specific task domain -- 2D planar block pushing with a top-down camera -- represents a narrow slice of the manipulation capabilities that production robots require.

Language-Table's most significant limitation is the extreme simplicity of its manipulation task. All 442,226 demonstrations involve pushing objects across a flat surface in 2D -- there is no grasping, no lifting, no placing, no insertion, no assembly, and no 3D manipulation of any kind. The action space is 2D planar end-effector velocity (x and y), meaning the robot arm only moves in the horizontal plane without controlling height, orientation, or gripper state. Production manipulation systems must operate in full 3D with grasping, precise placement, force-controlled assembly, and complex multi-step task sequences that Language-Table's task domain does not begin to address.

The sensor modality is a single top-down RGB camera capturing the tabletop from directly above. This fixed overhead viewpoint works well for planar block pushing but is unrepresentative of how production robots perceive their environment. Real robot systems use eye-in-hand cameras, wrist-mounted cameras, multi-view setups, depth sensors, force/torque sensors, proprioceptive streams, and tactile sensors. A policy trained exclusively on Language-Table's single top-down RGB view will not generalize to the camera configurations and multi-modal sensor stacks used in production deployments.

The language annotations, while impressively diverse (87,000 unique strings), were collected as real-time narrations during teleoperation. This means they carry the noise of spontaneous speech: grammatical inconsistencies, ambiguous references, and varying levels of specificity. For production systems where language instructions must be reliably parsed and executed, the annotation quality and consistency standards of Language-Table may not meet requirements. Production deployments typically need language annotations that have been validated against the demonstrated behavior by multiple annotators to ensure that the instruction unambiguously describes the task.

The object set is extremely limited. Language-Table demonstrations primarily involve colored blocks (red, blue, green, yellow) and a small number of toy objects (a toy banana, small figurines). These objects are all small, rigid, and lightweight -- selected specifically for the 2D pushing task. Production robots manipulate objects of widely varying size (millimeter-scale components to meter-scale packages), material (rigid, soft, deformable, fragile, wet), weight (grams to kilograms), and geometry (regular, irregular, articulated). The distribution of objects in Language-Table does not prepare a policy for this diversity.

Language-Table is the gold standard for research on language-conditioned manipulation and interactive language grounding. If your work focuses on how robots can follow open-vocabulary language instructions, how to design architectures for language-to-action mapping, or how to enable real-time interactive language control, Language-Table provides the largest, most carefully designed benchmark available. Its 87,000 unique language strings enable rigorous evaluation of language generalization, and the open-sourced simulation environment allows controlled experimentation.

As part of the OXE collection, Language-Table contributes valuable language-conditioned data to cross-embodiment pretraining. Generalist policies like Octo and OpenVLA benefit from Language-Table's language grounding -- even though the task is simple, the diversity of language instructions teaches models to map varied natural language to manipulation behavior. The Apache 2.0 license makes this pretraining use commercially permissible.

Switch to commercial data when your production system needs language-conditioned policies for real 3D manipulation tasks. The gap between pushing blocks on a flat surface and executing production manipulation workflows (pick-and-place, assembly, tool use, food preparation, warehouse fulfillment) is enormous. Production language-conditioned systems need demonstrations of complex tasks with validated language annotations, full sensor coverage, and the 6-DoF (or higher) action spaces that real manipulation demands.

The recommended pipeline is: pretrain on OXE (including Language-Table) for language-grounding priors and broad manipulation capability, then fine-tune on Claru's language-annotated, multi-modal demonstrations for your specific tasks and deployment environment. This captures Language-Table's strength in language understanding while grounding manipulation behavior in the task complexity and sensor coverage of real-world deployment.

Claru provides the language-conditioned manipulation data that bridges the gap between Language-Table's 2D block pushing and the 3D manipulation tasks that production robots perform. Where Language-Table captures 442,226 demonstrations of planar pushing with top-down RGB, Claru provides expert teleoperation demonstrations of full 6-DoF manipulation -- grasping, placing, assembly, insertion, tool use, and multi-step task sequences -- each paired with validated natural language task descriptions.

Every Claru demonstration includes the multi-modal sensor coverage that production policies require: multi-view RGB and depth imagery, 6-axis force/torque measurements, complete proprioceptive state, and gripper contact detection. Language annotations are not real-time narrations but carefully crafted task descriptions validated by multiple annotators for accuracy, specificity, and consistency. This ensures that the language-action mapping your policy learns is grounded in precise, unambiguous instructions.

Claru's task scope matches your production requirements, not the constraints of a tabletop pushing setup. Whether your language-conditioned robot needs to 'pick up the red box from the left shelf and place it in the shipping bin', 'insert the M4 screw into the threaded hole with 2 Nm torque', or 'slice the tomato into 5mm rounds' -- our demonstrations capture these tasks with the action dimensions, force profiles, and multi-step structure they require. The language vocabulary, task granularity, and instruction complexity are defined collaboratively to match your deployment's natural language interface.

All data ships with a commercial license and IP assignment, delivered in RLDS (for OXE co-training), HDF5, zarr, and LeRobot formats. Language annotations include both task-level descriptions and step-level sub-instructions where applicable, enabling hierarchical language-conditioned policy training at multiple temporal granularities.