CALVIN Alternative: Real-World Training Data for Production Robotics

CALVIN has been instrumental in advancing language-conditioned, long-horizon manipulation research. Its multi-environment evaluation protocol and sequential task chaining provide a uniquely demanding benchmark that has driven progress in combining language understanding with sustained robotic execution -- over 60 papers now cite CALVIN for evaluation. However, CALVIN's purpose is algorithmic benchmarking in simulation, not data for production training. The play data format, while innovative for learning diverse skills, produces trajectories that mix successful and incomplete actions without task-level ground truth until hindsight relabeling is applied. The simulated environments, while structurally varied across the A-D split, lack the visual complexity, physics fidelity, and sensor noise of real deployment settings. Policies that achieve high CALVIN scores routinely fail when transferred to real robots -- the sim-to-real gap in rendering, contact dynamics, and depth sensing is too large for direct transfer. Claru bridges these gaps by collecting task-directed demonstrations on your physical robot with natural language annotations from diverse human annotators. Our demonstrations are segmented by task with validated success outcomes, providing clean training signal for both single-step and multi-step policies. For teams that have developed their architecture on CALVIN and need to deploy on real hardware, Claru provides the real-world data -- with real depth noise, real contact physics, and real linguistic variety -- that makes the transition from benchmark performance to production reliability.

CALVIN (Composing Actions from Language and Vision) is a simulated benchmark for language-conditioned, long-horizon robot manipulation, developed by Oier Mees, Lukas Hermann, and Wolfram Burgard at the University of Freiburg. Published in IEEE Robotics and Automation Letters in 2022 (arXiv:2112.03227), CALVIN was designed to evaluate agents that can solve many manipulation tasks over extended horizons, specified only via natural language, from onboard sensor observations.

The benchmark includes approximately 24 hours of teleoperated unstructured play data together with 389 unique language instructions for 34 tasks across 4 manipulation environments (labeled A, B, C, D). These environments share structural similarities but differ in object positions, colors, and scene configurations, enabling evaluation of how well agents generalize across visual variations. Tasks include sliding, pushing, rotating, stacking, lifting, and using switches and drawers. The play data approach -- where operators perform free-form manipulation without specific task instructions -- was specifically designed to produce naturally diverse trajectories that cover multiple skills within single episodes, addressing the artificial uniformity of earlier task-directed collection protocols.

CALVIN's sensor suite is rich for a simulation benchmark: RGB-D images from both a static third-person camera and a gripper-mounted camera, proprioceptive information (joint angles, velocities), and vision-based tactile sensing. The simulation uses a Franka Emika Panda arm operating in a PyBullet-based physics environment. Each environment contains a tabletop scene with a sliding door, a drawer, a switch (LED light), a light switch, and several colored blocks -- providing sufficient object diversity for multi-task evaluation while remaining reproducible across research groups.

CALVIN's most distinctive contribution is its long-horizon evaluation protocol, which chains up to 5 tasks sequentially and measures cumulative success. An agent receives a sequence of language instructions (e.g., 'push the blue block right', 'turn on the light', 'open the drawer') and must execute them in order without resetting the environment between tasks. This evaluation structure tests whether policies can maintain coherent behavior across task boundaries -- a capability that single-task benchmarks cannot assess. The standard train-on-ABC / test-on-D protocol further tests generalization to unseen environment configurations. Released under the MIT License, CALVIN has become a standard evaluation platform adopted by over 60 research papers studying long-horizon, language-conditioned manipulation.

CALVIN's data collection protocol is based on unstructured play, a concept inspired by developmental psychology where operators interact freely with the environment without specific task goals. During collection, teleoperators explored the tabletop scene for extended periods, manipulating objects, operating mechanisms (drawers, switches), and performing sequences of actions that naturally span multiple task categories. This produces episodes where a single trajectory might contain pushing a block, opening a drawer, and toggling a switch -- all without explicit task boundaries.

The benchmark then applies hindsight language labeling: after collection, language annotators watch the recorded trajectories and retroactively assign natural language descriptions to segments where specific tasks were completed. This produces the 389 unique language instructions paired with trajectory segments, organized into 34 task categories. Each task category has approximately 11 unique language variations (e.g., 'push the red block left', 'slide the red block to the left', 'move the red block leftward'), though the linguistic variation is constrained by the template-based annotation protocol.

The four environments (A, B, C, D) share the same table and robot but differ in object colors, positions, and background textures. The standard evaluation protocol trains on environments A, B, and C and tests on D, measuring whether the policy generalizes to unseen visual configurations. This cross-environment evaluation is CALVIN's most demanding test and has proven to be a significant discriminator between methods -- many approaches that achieve high success on seen environments (A, B, C) drop substantially on the unseen environment D.

The long-horizon evaluation metric -- completing chains of 1 to 5 sequentially specified tasks -- has become CALVIN's signature contribution to the field. A score of '5/5' means the agent completed all 5 tasks in sequence. This metric exposed a critical weakness in early methods: many could complete individual tasks reliably but failed to maintain coherent state tracking across task boundaries, for example leaving the gripper in a state that prevented executing the next task. Methods like HULC (Mees et al., 2022), SuSIE (Black et al., 2024), and GR-1 have progressively improved on this metric, with current state-of-the-art approaches achieving average chain completion lengths above 3.5 out of 5.

CALVIN's fundamental limitation is the sim-to-real gap. All data exists in a PyBullet-based simulation with simplified rendering, idealized physics, and synthetic textures. Policies that score well on CALVIN's long-horizon evaluation chains (completing up to 5 tasks sequentially) routinely fail on physical hardware where lighting varies, objects have real material properties, and contact dynamics are stochastic. The visual gap is particularly severe: CALVIN's rendering produces uniform lighting, clean textures, and perfect edges that bear little resemblance to real camera footage with noise, reflections, and variable illumination.

The unstructured play data format, while valuable for learning diverse skills, creates challenges for production training. Play data mixes successful and incomplete actions within single trajectories, with no task-level segmentation or success labeling until hindsight relabeling is applied. The hindsight relabeling process itself is imperfect -- annotators may disagree on where a task starts and ends, and partially completed actions can be misclassified as successes or failures. Extracting clean, task-directed demonstrations from play data requires hindsight relabeling -- an active research area, not a solved engineering problem.

CALVIN's 34 tasks cover a narrow manipulation vocabulary focused on tabletop interactions: pushing, sliding, rotating, and basic pick-and-place. Production deployments typically require dozens to hundreds of task types including assembly, tool use, deformable object handling, and contact-rich manipulation that CALVIN does not address. The task set is also heavily biased toward rigid object manipulation in uncluttered scenes -- the tabletop contains only a few objects at a time, with ample free space between them.

The 389 language instructions, while diverse relative to earlier benchmarks, represent a limited linguistic distribution. Real users issue instructions with far greater variability, ambiguity, and context-dependence. Policies trained on CALVIN's instruction set overfit to its particular phrasings and struggle with natural language in deployment. The template-based annotation protocol means that instruction variation is largely syntactic rather than truly diverse -- multiple annotators might produce different word orders, but the underlying semantic templates are shared.

CALVIN's depth data is perfect synthetic depth with no noise, missing data, or sensor artifacts. Real depth sensors (Intel RealSense, Azure Kinect) fail on reflective surfaces, transparent objects, and at material boundaries -- challenges that simulation does not prepare policies to handle. Similarly, CALVIN's proprioception is noise-free joint state, while real robots exhibit encoder quantization, vibration artifacts, and calibration drift.

CALVIN is the right choice when evaluating long-horizon, language-conditioned manipulation architectures. Its structured train-on-ABC / test-on-D protocol provides a clean generalization benchmark, and its multi-task sequential evaluation measures whether models maintain coherent behavior across task chains -- a capability that most benchmarks cannot evaluate. If you are publishing a paper on long-horizon planning, CALVIN provides the standardized evaluation that reviewers expect and that enables fair comparison with prior work.

CALVIN is also useful for studying how agents learn from unstructured play data without explicit task labels. If your research explores hindsight relabeling, skill discovery from play, or self-supervised task decomposition, CALVIN's play data format is specifically designed for these investigations. The availability of both annotated segments and raw play data enables controlled experiments on the value of language supervision.

Switch to Claru when you need language-conditioned manipulation policies that work on physical robots. Real-world language-conditioned systems must handle real visual complexity, real contact dynamics, and the full breadth of natural language variation. Claru collects task-directed demonstrations on your robot with natural language annotations from multiple annotators, producing the grounded language-action correspondence that simulation cannot provide. The transition from CALVIN to real-world deployment is not a fine-tuning step -- it requires fundamentally different training data that captures the physics, visual complexity, and linguistic diversity of real environments.

Claru provides the real-world grounding that transforms CALVIN-developed architectures into deployable products. Where CALVIN offers simulated play data with template-based language instructions, Claru collects task-directed demonstrations on your physical robot with diverse natural language annotations generated by multiple human annotators describing the same behavior in their own words. This annotation diversity produces the robust language grounding that deployment requires.

For teams that have validated their long-horizon planning approach on CALVIN, Claru delivers the fine-tuning data that closes the sim-to-real gap. Our demonstrations capture real depth sensor noise, real contact dynamics, and real environmental variability -- the factors that determine whether a long-horizon policy can maintain coherent execution across multiple real-world task steps. The CALVIN-to-deployment transition is not just about visual domain adaptation; it requires learning how real objects behave when grasped, pushed, and stacked in ways that simulation physics cannot accurately model.

Claru also scales beyond CALVIN's 34-task vocabulary. We collect demonstrations for whatever manipulation tasks your deployment requires, including contact-rich operations, deformable objects, and tool use that CALVIN's simulation cannot support. Every demonstration is validated for task success and annotated with natural language at the diversity level real-world language conditioning demands.

Data is delivered in RLDS, HDF5, zarr, or LeRobot format with standardized schemas compatible with the same training pipelines used for CALVIN-based research. For teams using HULC, SuSIE, or other CALVIN-native architectures, we provide data in the exact episode format these methods expect, with task segmentation, language pairing, and observation structure that enables direct integration without preprocessing pipeline changes.