Real-World Data for DexArt
DexArt benchmarks dexterous manipulation of everyday mechanisms with a 16-DOF Allegro hand. Real-world data captures the mechanical variation that simulation cannot parametrize.
DexArt at a Glance
DexArt Task Suite
Each task requires different dexterous strategies and presents distinct sim-to-real challenges.
| Task | Mechanism Type | Key Manipulation Skill | Real-World Challenge |
|---|---|---|---|
| Turn Faucet | Revolute joint (handle rotation) | Rotational torque with stable grip | Valve type variation (ball, compression, cartridge) |
| Open Laptop | Revolute joint (hinge) | Upward lift maintaining edge contact | Hinge stiffness, screen weight, magnetic closure |
| Lift Bucket | Revolute joint (handle arc) | Grasp and lift with weight compensation | Handle material flex, load-dependent dynamics |
| Flip Toilet Seat | Revolute joint (soft-close hinge) | Controlled rotation past gravity threshold | Soft-close dampers, spring-loaded mechanisms |
DexArt vs. Related Dexterous Benchmarks
| Feature | DexArt | Real Robot Challenge | DexMV | In-Hand Reorientation |
|---|---|---|---|---|
| Object type | Articulated mechanisms | Rigid cubes | Rigid objects | Rigid objects |
| Hand model | Allegro (16-DOF) | TriFinger (9-DOF) | Shadow Hand (24-DOF) | Allegro (16-DOF) |
| Key skill tested | Mechanism operation | Cube repositioning | Multi-finger grasping | In-hand rotation |
| Observation | Point cloud | RGB + state | RGB video | Proprioception + tactile |
| Generalization test | Intra-category objects | Cube pose variation | Object diversity | Novel objects |
Benchmark Profile
DexArt is a benchmark for dexterous manipulation of articulated objects using multi-finger robotic hands. Presented at CVPR 2023 by Bao et al., it evaluates policies on tasks requiring coordinated finger movements to operate real-world mechanisms — faucets, laptops, buckets, and toilet seats — using a simulated 16-DOF Allegro hand in the SAPIEN physics engine.
The Sim-to-Real Gap
DexArt uses SAPIEN for articulated object physics but real mechanical assemblies have unique friction profiles, backlash, and resistance patterns that vary over their range of motion. The simulated Allegro hand model simplifies tendon routing, actuator compliance, and fingertip deformation. Real dexterous manipulation of faucets and laptop lids involves material interactions — rubber gaskets, spring-loaded hinges, hydraulic dampers — that parametric simulation cannot fully capture.
Real-World Data Needed
Real-world dexterous manipulation of articulated objects with multi-finger hands or human hands. Tactile and force data during mechanism operation showing how grip force and finger placement adapt to resistance. Diverse mechanical variation across the same object category — different faucet valve types, laptop hinge stiffnesses, bucket handle materials.
Complementary Claru Datasets
Custom Dexterous Articulated Object Collection
Purpose-collected data operating real faucets, laptops, and mechanisms with multi-finger coordination and force measurements captures the adaptive grip strategies DexArt evaluates.
Egocentric Activity Dataset
Human hand manipulation of household mechanisms across 100+ locations provides demonstrations of adaptive, force-sensitive coordination with naturally varying mechanical resistance.
Manipulation Trajectory Dataset
Diverse contact-rich manipulation recordings provide broader understanding of object-hand interaction dynamics that transfers to dexterous articulated manipulation tasks.
Bridging the Gap: Technical Analysis
DexArt addresses the intersection of two hard problems: dexterous manipulation and articulated object interaction. Each problem alone is challenging; together they create a high-dimensional contact space where finger placement, force application, and timing must coordinate precisely with object mechanism dynamics.
The faucet task illustrates the challenge. Turning a real faucet handle requires gripping the handle (contact planning), applying rotational force (torque control), and following the mechanism's arc (trajectory following) — all with fingers that must maintain contact despite the handle's changing position and orientation. In simulation, friction is constant and the mechanism arc is perfectly known. On real hardware, friction varies with grip position, the mechanism has backlash and resistance peaks, and the hand must adapt in real-time to unexpected forces.
DexArt's use of PartNet-Mobility objects provides geometric diversity but not mechanical diversity. All simulated faucets use the same friction model and joint dynamics parametrized by a few constants. Real faucets range from smooth ball valves to stiff compression valves, each requiring fundamentally different grip strategies and force profiles. A ball valve needs gentle rotational force; a compression valve needs strong downward push combined with rotation.
The point cloud observation space is a deliberate design choice that avoids the visual sim-to-real gap but introduces a 3D perception challenge. Real depth sensors produce noisy, incomplete point clouds with occlusions from the hand itself. Policies trained on clean simulated point clouds must handle these artifacts when deployed with real depth cameras.
Human hand manipulation of these everyday mechanisms provides the richest training signal for dexterous policies. Claru's egocentric data captures humans operating faucets, laptops, and household mechanisms naturally — providing demonstrations of the adaptive, force-sensitive manipulation that DexArt evaluates.
Key Papers
- [1]Bao et al.. “DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects.” CVPR 2023, 2023. Link
- [2]Chen et al.. “Visual Dexterity: In-Hand Reorientation of Novel Objects.” ICRA 2023, 2023. Link
- [3]Shaw et al.. “Learning Dexterous Manipulation from Human Demonstrations.” CoRL 2023, 2023. Link
- [4]Qi et al.. “General In-Hand Object Rotation with Vision and Touch.” CoRL 2023, 2023. Link
Frequently Asked Questions
It combines two hard problems: multi-finger coordination (16 DOF hand with 4 independent fingers) and articulated object dynamics (mechanisms with joints, friction, and backlash). The contact space is enormous — small changes in finger placement dramatically affect whether a mechanism operates smoothly or jams. Each contact point has its own friction cone, and all must be coordinated simultaneously.
DexArt simulates faucets, laptops, and mechanisms with parameterized physics using constant friction and damping coefficients. Real versions of these objects have unique resistance profiles that vary over their range of motion — a faucet may be stiff at the start and loose in the middle. A policy trained on DexArt's constant-parameter faucets may fail on a real faucet with a non-linear resistance curve.
Yes. Human hand manipulation of everyday mechanisms demonstrates adaptive, force-sensitive strategies that transfer to robot dexterous control. Visual demonstrations show finger placement selection, force modulation over mechanism travel, and recovery from partial slips. These strategies, once learned from human data, can be adapted to specific robot hand kinematics through retargeting or representation learning.
DexArt trains on a subset of object instances within each category (e.g., 5 faucet models) and evaluates on held-out instances from the same category. This tests whether the policy learns general dexterous strategies for operating faucets — regardless of handle shape — or memorizes object-specific grip sequences that fail on novel geometries.
Point clouds provide 3D geometric information that is more directly useful for planning finger placements and grasp strategies. They also reduce the visual sim-to-real gap by providing geometry rather than appearance. However, real depth sensors produce noisier, sparser point clouds with occlusions from the hand itself, creating a perception gap that clean simulated point clouds do not expose.
Get Dexterous Manipulation Data
Discuss real-world dexterous manipulation data for articulated object benchmarks with authentic mechanical variation.