Human-Robot Handover Training Data

Claru collects human-robot handover data with calibrated multi-sensor rigs capturing both robot state and human motion simultaneously. Our participant recruitment process ensures minimum 30 unique participants per campaign with demographic diversity tracking. We maintain a standardized object kit of 30-50 items organized by handover strategy type (symmetric, handled, fragile, heavy, dangerous) with full physical metadata. Annotations include precise timing labels (initiation, grasp-establishment, release), force/torque load-transfer profiles, object orientation correctness checks, and human preference ratings on naturalness, comfort, safety, and timing — all collected through a structured in-session rating protocol. We deliver in RLDS, HDF5, or custom formats with full sensor calibration, participant demographics (anonymized), object metadata, and stratified splits by participant, object category, and handover direction.

Human-robot handover — the act of transferring an object between a human hand and a robot gripper — is one of the most fundamental capabilities for collaborative robots (cobots) deployed alongside humans. Every assistive, manufacturing, surgical, or service robot that operates in human environments must give objects to people and receive objects from them. A warehouse cobot passing a tool to a technician, a surgical assistant handing an instrument to a surgeon, a home robot delivering a glass of water — all require fluent handover behavior. The market for collaborative robots (cobots) is projected to exceed $12 billion by 2030, and handover capability is a prerequisite for most cobot applications.

The difficulty lies in the interaction dynamics. Unlike pick-and-place manipulation where the robot acts alone, handover involves a reactive partner: the human. The robot must predict when the human is ready to receive (or give) the object, position the object where the human can comfortably grasp it, release at the right moment (too early drops the object, too late feels like a tug-of-war), and adapt to the human's reaching speed, grip preferences, and body posture. Human handover behavior is remarkably nuanced — we unconsciously orient a mug with the handle toward the receiver, hand a knife handle-first, and adjust our release timing based on whether the receiver has established a firm grip. Replicating this social intelligence in a robot requires training data that captures the full spectrum of human handover behavior.

The data challenge is twofold. First, handover is inherently a two-agent interaction, requiring synchronized capture of both human motion (hand pose, body pose, gaze direction) and robot state (joint positions, gripper force, end-effector trajectory). Second, handover preferences are culturally and individually variable: preferred handover height, approach direction, and timing differ between people, and the 'right' behavior depends on the object (fragile items handed slowly, heavy items with clear load-transfer signaling). Training data must cover sufficient participant diversity to prevent the policy from overfitting to a single person's handover style.

Early handover systems used hard-coded rules: detect that a human hand is within a distance threshold, wait for grip force above a threshold, then release. These reactive systems are slow and awkward — the robot holds the object rigidly until the human grabs it, then abruptly releases, producing a jerky, unnatural exchange. The HandoverSim benchmark (Chao et al., CVPR 2022) formalized evaluation metrics for handover: success rate, handover duration (time from human reach initiation to complete transfer), and human effort (how much the human must adjust their reach). Their results showed that even well-tuned reactive controllers achieve handover durations 2-3x longer than human-human handovers.

Learning-based approaches have progressed through three generations. Prediction-based methods (Yang et al., ICRA 2021) learn to predict the human's reaching trajectory from early motion cues (the first 200-500 ms of arm motion), enabling the robot to pre-position the object at the predicted receiving location before the human's hand arrives. These methods require 2,000-5,000 handover recordings with full body pose tracking to learn reaching trajectory prediction. Reactive force-based methods (Rosenberger et al., RA-L 2021) use force/torque sensing to detect when the human has established a grip and modulate the robot's grip force smoothly during the transfer phase, producing a more natural shared-support period rather than an abrupt release. The current state of the art combines both: predict the human's intent and pre-position the object, then use force feedback to execute a smooth load transfer.

The Yang et al. (CoRL 2022) study on socially-aware handover demonstrated that humans prefer robots that orient objects for easy grasping (handle toward receiver), maintain eye-level presentation height, and approach from the front-right for right-handed receivers. Their policy, trained on 3,000 handover demonstrations with post-interaction human preference ratings, achieved a 4.2/5.0 naturalness score compared to 2.8/5.0 for a baseline reactive controller. The key data insight: handover quality requires not just success/failure labels but human subjective ratings of comfort, naturalness, and perceived safety. These preference annotations are expensive but essential for training socially fluent handover behavior.

Multi-object and context-dependent handover is an emerging frontier. Handing a full coffee mug requires different orientation, speed, and grip than handing a pencil or a heavy wrench. Current datasets cover 10-30 object types, but production deployment in diverse environments requires handling hundreds of object categories with varying mass (1 g to 5 kg), fragility, sharp edges, and temperature. The training data must pair each object type with the appropriate handover strategy, including warnings for dangerous objects (sharp, hot, heavy) communicated through robot motion hesitation or verbal cues.

Handover data collection requires capturing both the robot and the human partner simultaneously, at sufficient resolution to model the interaction dynamics. The robot-side sensor stack includes: wrist-mounted RGB-D camera (for close-up view of the handover zone), proprioceptive data (joint positions and velocities at 50-100 Hz), and a 6-axis force/torque sensor at the wrist (for grip force and load transfer detection at 100-500 Hz). The human-side sensor stack includes: hand tracking (MediaPipe, Leap Motion, or marker-based OptiTrack for sub-millimeter accuracy), body pose estimation (from RGB cameras using SMPL-based methods or motion capture suit), and optionally eye gaze tracking (Tobii Pro Glasses or Pupil Labs) to capture attentional cues that predict handover intent.

Participant recruitment is critical. Handover behavior varies with age, height, hand size, handedness, and cultural background. Datasets collected with only lab members (typically 5-10 young, tech-savvy adults) produce policies that fail on elderly users, children, or people with motor impairments. Claru's handover collection protocol requires a minimum of 30 unique participants spanning: age range (18-75), gender balance, left- and right-handed (minimum 20% left-handed), standing height range (150-190 cm), and no more than 3 participants from any single demographic group. Each participant performs 50-100 handover exchanges (both giving and receiving) with the robot, producing 1,500-3,000 episodes per collection campaign.

The object set must span the physical diversity of real handover scenarios. We use a standardized object kit of 30-50 items organized by handover strategy type: symmetric objects (balls, cylinders — no orientation preference), handled objects (mugs, tools, scissors — handle orientation matters), fragile objects (eggs, glass cups — require slow approach and gentle grip), heavy objects (1-5 kg weights, books — require shared support phase), and dangerous objects (scissors, knives — require handle-first presentation and verbal warning). Each object is labeled with mass, center-of-mass location, preferred grasp type, and handover orientation rules.

Annotation captures both the physical dynamics and the social quality of each handover. Physical annotations include: handover initiation timestamp (when the giver begins extending the object), grasp-establishment timestamp (when the receiver's grip reaches sufficient force), release-completion timestamp (when the giver fully releases), load transfer profile (force over time during the shared-support phase), and success/failure with failure taxonomy (dropped, human refusal to reach, collision, timeout). Social quality annotations are collected post-episode: the human participant rates each handover on a 1-5 scale for naturalness, comfort, perceived safety, and timing appropriateness. These ratings provide the reward signal for training preference-aligned handover policies.