RLBench Alternative: Real-World Training Data for Production Robotics

RLBench established the template for modern robot learning benchmarks: diverse tasks, rich multi-view observations, and language conditioning. Its 100-task suite pushed the community to develop manipulation architectures that generalize across skill categories, and models like PerAct, RVT, and Act3D were born from RLBench evaluation. But RLBench's purpose is algorithmic comparison in simulation, not data for production training. The CoppeliaSim rendering pipeline, motion-planned trajectories, and single-scene environment create a significant gap between benchmark performance and real-world deployment. Claru closes this gap by providing real-world demonstrations that preserve the observation structure your RLBench-developed architecture expects. We configure multi-view camera setups matching your model's input geometry, collect expert human demonstrations that capture natural manipulation dynamics, and record with real sensors whose noise characteristics your policy must learn to handle. Teams that develop on RLBench and deploy with Claru data get the best of both worlds: rapid algorithmic iteration in simulation followed by production-grade performance from real-world fine-tuning. Our data is delivered in formats compatible with your existing training pipeline, with commercial licensing that clears the path to deployment.

RLBench is a large-scale benchmark and learning environment for robot manipulation, developed by Stephen James, Zicong Ma, and Andrew Davison at Imperial College London's Dyson Robotics Lab. First published in 2020, RLBench provides 100 unique manipulation tasks in the CoppeliaSim (V-REP) simulator, ranging from simple reaching and picking to complex multi-step activities like opening jars, sorting shapes, stacking cups, and operating switches. Each task includes procedurally generated demonstrations via motion planning, with support for generating unlimited additional demonstrations.

RLBench's observation space is notably rich for a simulation benchmark. Each timestep provides multi-view RGB-D images from four cameras (front, left shoulder, right shoulder, and wrist), full proprioceptive state (joint positions, velocities, forces), end-effector pose, gripper state, and a natural language task description. The multi-view setup was designed to mirror the camera configurations used on real robot systems, and the structured observation space made RLBench a popular testbed for multi-view fusion architectures.

The benchmark uses a simulated Franka Emika Panda arm with a Franka Hand (parallel-jaw gripper) in CoppeliaSim. Tasks are defined as Python classes with customizable parameters (object positions, orientations, counts), allowing researchers to generate diverse configurations. RLBench supports both reinforcement learning and imitation learning workflows, and its motion-planned demonstrations provide reliable expert trajectories for behavioral cloning.

Released under the MIT License, RLBench became one of the most cited robotics simulation benchmarks, influencing subsequent work on language-conditioned manipulation (PerAct, RVT, Act3D), multi-view policy learning, and 3D manipulation representations. Its 100-task diversity -- much broader than earlier benchmarks that typically offered 5-10 tasks -- established the expectation that robot learning benchmarks should evaluate generalization across a wide range of skills.

RLBench's demonstrations are generated by motion planners, not by humans. Motion-planned trajectories are geometrically optimal but kinematically unnatural -- they take straight-line paths in joint space, lack the smooth acceleration profiles of human manipulation, and do not exhibit the adaptive micro-corrections that skilled operators make during contact. Policies trained on motion-planned demonstrations learn fundamentally different manipulation strategies than those trained on human demonstrations, and the gap matters for tasks requiring dexterity or compliance.

The sim-to-real gap is substantial for RLBench due to CoppeliaSim's rendering and physics limitations. Rendered RGB images look distinctly synthetic -- flat textures, uniform lighting, no specular highlights or subsurface scattering. Real depth sensors produce noisy, incomplete depth maps (especially on reflective or transparent surfaces), while RLBench provides perfect synthetic depth. Policies that rely on depth inputs trained in RLBench overfit to this idealized signal and fail with real sensor data.

RLBench uses a single robot -- the simulated Franka Panda -- with a fixed workspace geometry. The workspace is a single tabletop scene with a limited backdrop. There is no environmental diversity: no clutter, no dynamic lighting changes, no competing objects, no background distractors. Real production environments are dramatically more complex, and policies trained in RLBench's clean setting are fragile when confronted with real-world visual complexity.

Task descriptions in RLBench are single template strings per task (e.g., 'put the lid on the pot'). Language-conditioned policies trained on these templates do not learn to handle the variability of real natural language instructions, where users might say 'cover the pot', 'close it up', or 'put the cover back on' to mean the same thing.

While RLBench can generate unlimited demonstrations via motion planning, more does not always mean better. The demonstrations lack the diversity that comes from different human operators, different approach strategies, and different error-recovery behaviors. This limits the robustness of policies trained exclusively on RLBench data, particularly for tasks where multiple viable strategies exist.

RLBench remains the gold standard for evaluating language-conditioned 3D manipulation architectures. If you are developing or comparing models like PerAct, RVT, or Act3D, RLBench's 100-task suite provides the standardized evaluation that the community expects. Its multi-view RGB-D observations and language conditioning make it particularly well-suited for methods that build explicit 3D representations for manipulation.

RLBench is also valuable for rapid architecture exploration. Because demonstrations are generated by motion planning, you can create unlimited training data for any of the 100 tasks without hardware overhead. This makes it ideal for hyperparameter sweeps, ablation studies, and initial architecture validation before committing to real-world data collection.

Move to commercial data when deployment is the goal. RLBench's simulated Franka in a clean tabletop scene does not prepare a policy for the visual complexity, sensor noise, and physical dynamics of real deployment environments. Claru collects demonstrations from human teleoperators on your physical robot, ensuring that the training data reflects the exact conditions your policy will face in production.

Many teams follow a three-phase approach: develop the architecture on RLBench for rapid iteration, validate on a small real-world pilot with Claru data, then scale collection for production deployment. RLBench handles phase one efficiently; Claru handles phases two and three.

Claru transforms RLBench-validated architectures into production-ready systems by providing the real-world data these models need for deployment. If you have developed a multi-view manipulation policy on RLBench, Claru collects demonstrations with a matching multi-camera setup on your physical robot, with calibrated extrinsics so your model can directly consume real-world observations in the same format it was designed for.

Where RLBench provides motion-planned demonstrations, Claru provides human demonstrations that capture the natural manipulation strategies, force modulation, and error recovery behaviors that motion planners cannot generate. These human-like demonstrations train more robust policies, especially for contact-rich tasks like insertion, wiping, or tool use where compliance and adaptation are essential.

Claru's language annotations are generated by multiple annotators describing the same demonstration in their own words, producing the linguistic diversity needed for language-conditioned policies to generalize beyond template instructions. Every annotation is validated for accuracy against the demonstrated behavior, ensuring the language-action correspondence is reliable.

We deliver in the observation format your architecture expects: multi-view RGB-D with camera intrinsics and extrinsics, proprioception at your control frequency, and optional force/torque and tactile streams. For teams transitioning from RLBench, our data schema is designed to be a drop-in replacement that maintains the observation structure while replacing synthetic signals with real ones.