Hand-Object Interaction: Datasets, Annotation, and Training Data for Dexterous Robotics

Hand-object interaction (HOI) is a multidisciplinary research area at the intersection of computer vision, biomechanics, and robotics that studies how human hands physically engage with objects. An HOI dataset captures the spatial, temporal, and force-based relationships between hand surfaces and object surfaces during manipulation — from the initial reach and contact establishment through the grasp, in-hand adjustment, functional use, and release phases of object handling.

At the vision level, HOI encompasses detection (identifying interacting hand-object pairs in images), recognition (classifying the interaction type), and reconstruction (recovering full 3D hand and object meshes with contact maps). Detection benchmarks like HICO-DET (Chao et al., 2018) contain over 150,000 annotated human-object pairs spanning 600 interaction categories. Reconstruction datasets like GRAB (Taheri et al., 2020) provide mocap-quality 3D hand and body meshes during natural grasping of 51 everyday objects, captured from 10 subjects.

For robotics, the critical value of HOI data lies in demonstrations of dexterous manipulation strategies. Human hands solve manipulation problems that robots struggle with: adjusting grip mid-task, using fingertip sensitivity to detect slip, exploiting gravity and surface friction as manipulation aids. HOI datasets encode these strategies implicitly in the joint trajectories and contact sequences. Methods like DexMV and DexArt retarget human hand data to robot hands, using HOI demonstrations as the supervision signal for dexterous policy learning.

The field distinguishes between egocentric and exocentric HOI data. Egocentric data, captured from head-mounted cameras, provides the first-person viewpoint that matches how an embodied agent perceives its own manipulation. Datasets like Ego4D, Epic-Kitchens, and H2O capture thousands of hours of egocentric manipulation. Exocentric data from fixed or third-person cameras captures the full spatial context of the interaction, including body posture and workspace layout. The most informative HOI datasets combine both viewpoints with synchronized 3D hand tracking, enabling models to learn viewpoint-invariant manipulation representations.

Building an effective HOI dataset for robotics requires careful decisions about capture methodology, annotation granularity, and downstream compatibility. The capture setup must balance tracking accuracy against naturalistic behavior — mocap gloves with markers achieve millimeter-level hand pose accuracy but alter how people grasp small or flexible objects, while markerless vision-based tracking preserves natural behavior at the cost of 5-15mm pose error during heavy occlusion.

Annotation granularity should match the intended use case. For affordance learning (predicting where to grasp an object), coarse annotations of contact regions and grasp type are sufficient and can be produced at 5-10 annotations per minute. For hand-to-robot retargeting, per-frame 3D hand joint positions and per-vertex contact labels are needed, requiring either automated tracking with manual correction or full mocap — annotation throughput drops to 1-3 minutes of video processed per hour of annotator time. For action recognition, temporal phase boundaries (reach, contact, manipulate, release) and verb-noun action labels are the minimum requirement.

Object diversity is the single most impactful factor for generalization. HOI datasets with 50 or fewer objects tend to overfit to specific geometries. The OakInk dataset demonstrated that expanding to 1,800 objects across shape and material categories dramatically improved cross-object generalization in downstream grasp prediction models. For production datasets, Claru targets a minimum of 200 unique objects per domain (kitchen, workshop, warehouse), stratified across size, shape, weight, and material categories.

Data format compatibility determines integration cost. The standard format for 3D HOI data is MANO hand model parameters (pose and shape vectors) plus rigid object 6-DOF pose per frame, stored in HDF5 or pickle files. For video-based HOI, the Ego4D and Epic-Kitchens formats (MP4 video + JSON temporal annotations) are widely adopted. Claru delivers HOI datasets with MANO-compatible hand representations, object mesh registrations, and temporal annotations in formats that plug directly into DexMV, DexArt, and custom imitation learning pipelines.

Claru's data collection infrastructure is purpose-built for capturing the diverse, real-world hand-object interaction data that dexterous robotics demands. Our 10,000+ collectors across 100+ cities record egocentric manipulation video in authentic environments — kitchens, warehouses, retail floors, and workshops — using standardized camera rigs with depth sensing. This geographic and environmental diversity produces HOI data spanning thousands of unique objects, manipulation strategies, and interaction contexts that single-lab captures cannot match. Claru's annotation pipeline labels hand keypoints, object poses, contact states, grasp taxonomy classes, and temporal action phases, with multi-annotator redundancy enforcing consistency. For teams building dexterous robot policies via hand-to-robot retargeting or visual affordance learning, Claru provides the volume and variety of human demonstration data that bridges the gap between lab prototypes and production deployment.