ManiSkill Alternative: Real-World Training Data for Production Robotics

ManiSkill has advanced the state of the art in generalizable manipulation research through its GPU-parallelized simulation and diverse task library. Its ability to run 100,000+ parallel environments makes it unmatched for reinforcement learning experimentation and rapid algorithm iteration. However, the path from simulation benchmark to production deployment inevitably requires real-world data. Policies that generalize across ManiSkill's synthetic object instances often fail to generalize across real-world object variants, where visual appearance, material properties, and contact dynamics diverge from simulation. Claru bridges this gap by collecting teleoperated demonstrations on your physical robot, in your deployment environment, with real objects from your production workflow. Our data captures the contact physics, sensor noise, and visual complexity that GPU simulation cannot faithfully model. Teams that have validated their approach on ManiSkill can use Claru's real-world demonstrations for the critical fine-tuning step that converts a research prototype into a deployable system. We deliver in standard formats (RLDS, HDF5, zarr, LeRobot) with multi-modal sensor streams -- including force/torque and tactile data that ManiSkill does not provide -- enabling you to train policies that are robust to the full complexity of real-world manipulation.

ManiSkill is a family of GPU-parallelized simulation benchmarks for generalizable robotic manipulation, developed by Hao Su's lab at UC San Diego in collaboration with Hillbot. The project has evolved through three major releases: ManiSkill1 (2021) focused on object manipulation with point cloud observations, ManiSkill2 (2023) expanded to 20 task families with flexible observation modes, and ManiSkill3 (2024) introduced massive GPU parallelization using the SAPIEN simulator, enabling up to 100,000+ parallel environments on a single GPU for reinforcement learning at unprecedented throughput.

ManiSkill3 represents the most significant iteration, offering over 20 task categories spanning rigid-body manipulation (pick-and-place, stacking, peg insertion), articulated-object interaction (opening cabinets, drawers), soft-body manipulation, and mobile manipulation. The benchmark leverages PartNet-Mobility and the YCB object dataset to provide thousands of 3D object models with realistic geometries, enabling evaluation of how well policies generalize across object instances. Observations include RGB-D images from configurable camera viewpoints, point clouds, and full proprioceptive state.

A key engineering contribution of ManiSkill3 is its GPU-accelerated rendering and physics pipeline built on SAPIEN and PhysX 5. This allows researchers to generate millions of simulation frames per hour for reinforcement learning, making it one of the fastest robotics simulation benchmarks available. The benchmark supports both RL and imitation learning workflows, with demonstration datasets generated via motion planning or teleoperation within the simulator.

ManiSkill is released under the Apache 2.0 license and has become a standard evaluation platform for manipulation research, particularly for methods that aim to generalize across object geometries. It has been used to evaluate visual RL algorithms, point-cloud-based policies, and sim-to-real transfer pipelines. The ManiSkill Challenge series has attracted hundreds of submissions from research groups worldwide.

ManiSkill's primary value -- massive GPU-parallelized simulation -- is also the source of its core limitation for production robotics: the sim-to-real gap. While PhysX 5 provides fast rigid-body simulation, it does not faithfully model the nuances of real-world contact: surface friction varies with wear, deformable materials compress unpredictably, and compliant joints exhibit backlash that rigid-body simulators do not capture. Policies trained in ManiSkill that achieve 90%+ success rates in simulation often drop to 40-60% on physical hardware without significant real-world fine-tuning.

Object diversity in ManiSkill comes from 3D mesh databases like PartNet-Mobility and YCB. These are geometrically accurate but lack the visual variability of real objects -- worn labels, transparent packaging, reflective surfaces, and deformable materials are either absent or poorly approximated. For production applications involving consumer products, food items, or industrial components, the visual domain gap between ManiSkill's rendered objects and real counterparts is substantial.

ManiSkill's robot selection is limited to a fixed set of simulated platforms. Teams deploying custom end-effectors, proprietary arm configurations, or robots not in ManiSkill's library cannot directly use the benchmark. Even for supported robots like the Franka, the simulated version omits real-world characteristics like joint friction, cable routing interference, and calibration drift.

Sensor modeling in ManiSkill provides clean RGB-D and point clouds without the noise profiles of real depth sensors (structured-light artifacts, reflective-surface failures, range limitations). Force/torque and tactile sensing are not supported, ruling out training for contact-rich tasks like insertion, polishing, or packing where haptic feedback drives policy decisions.

Finally, ManiSkill's environments are modular but synthetic. Real production environments have cluttered backgrounds, variable lighting (fluorescent, natural, mixed), and dynamic obstacles that are difficult to capture through domain randomization alone. The controlled nature of simulation environments creates policies that are brittle when faced with the uncontrolled variability of real deployment sites.

ManiSkill excels in three research scenarios. First, for reinforcement learning at scale: its GPU parallelization enables RL training that would be impossibly slow on physical hardware. If your research question is about sample efficiency, reward shaping, or policy architecture for RL, ManiSkill provides the throughput needed for meaningful experiments. Second, for object generalization research: the large 3D model library lets you test whether policies transfer across object geometries in a controlled setting. Third, for rapid prototyping of manipulation algorithms where you need to iterate on ideas before committing to expensive real-world data collection.

Transition to real-world data when you have a deployment target. Once you know your robot platform, your task set, and your deployment environment, the return on simulated data diminishes sharply. Each hour of real-world data collected in your specific setting closes the sim-to-real gap more effectively than days of simulated training with domain randomization. Claru collects demonstrations that exactly match your production conditions.

Many teams find the optimal path is staged: use ManiSkill for early-stage algorithm development and RL pretraining, then transition to Claru's real-world data for fine-tuning and production readiness. This leverages ManiSkill's speed for exploration while relying on real data for the final performance push that simulation cannot provide.

Claru provides the real-world data layer that ManiSkill's simulation pipeline cannot generate. Where ManiSkill offers synthetic demonstrations from a physics engine, Claru deploys trained teleoperators with your physical robot in your actual environment. The resulting data captures true contact dynamics, real sensor noise, and the visual complexity of your production setting -- all the elements that simulation approximates but cannot replicate.

For teams using ManiSkill's RL-pretrained policies, Claru's demonstrations serve as the fine-tuning dataset that bridges simulation to reality. Research has consistently shown that even a relatively small real-world dataset (hundreds to low thousands of demonstrations) can dramatically improve the real-world performance of sim-pretrained policies, often more than 10x more simulated data would achieve.

Claru also covers the modalities ManiSkill lacks. If your task requires force/torque feedback for insertion, tactile sensing for deformable object manipulation, or high-frequency proprioceptive logging for dynamic movements, we capture these streams synchronized with visual data. Every demonstration passes multi-stage quality control and is delivered in RLDS, HDF5, zarr, or LeRobot format compatible with your existing training pipeline.

Beyond data collection, Claru provides ongoing expansion. As your deployment requirements evolve -- new tasks, new environments, new object categories -- we scale collection accordingly. This continuous data flywheel replaces ManiSkill's static benchmark releases with a living dataset that grows with your product.