Training Data for HumanPlus

Claru provides data for all three stages of the HumanPlus pipeline. For HST training (Stage 1), we deliver human motion capture data in SMPL/SMPL-H format from optical motion capture systems (Vicon, OptiTrack) at 120 Hz, covering household manipulation, industrial tasks, and daily activities that extend AMASS's coverage. For HIT data collection (Stage 2), we provide the teleoperation infrastructure and trained operators to collect shadowing demonstrations on Unitree H1 and other humanoid platforms, delivering 20-50 high-quality demonstrations per task with synchronized egocentric RGB and 33-DoF joint trajectories at 30 Hz. For visual backbone pretraining extensions, our catalog of 3M+ egocentric human activity videos provides rich visual data from head-mounted cameras. All data includes task success labels, camera calibration, and format compatibility verification with HumanPlus's open-source codebase on GitHub.

HumanPlus is a full-stack system for humanoid robots to learn motion and autonomous skills from human data, developed at Stanford by Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Published at CoRL 2024 (arXiv 2406.10454), HumanPlus demonstrates that a humanoid robot can learn to shadow human motion in real time using only a single RGB camera, then leverage that shadowing capability to collect task-specific demonstration data and learn autonomous skills via behavior cloning.

The system runs on a custom 33-DoF humanoid built on the Unitree H1 platform, featuring two 6-DoF dexterous hands (Inspire Hands), two 1-DoF wrists, and a 19-DoF body (two 4-DoF arms, two 5-DoF legs, and a 1-DoF waist). Two egocentric RGB cameras are mounted on the head for visual perception. Using up to 40 teleoperated demonstrations per task, the humanoid autonomously completes tasks including wearing a shoe and standing up, unloading objects from warehouse racks, folding a sweatshirt, rearranging objects, typing on a keyboard, and greeting another robot -- with success rates ranging from 60% to 100%.

The fundamental insight of HumanPlus is that the human body is the best teleoperation interface for a humanoid robot. Rather than designing complex teleoperation rigs with haptic gloves or VR controllers, HumanPlus uses off-the-shelf pose estimation to track a human operator's body motion from a single RGB camera, retargets that motion to the humanoid's joint space in real time, and uses the resulting data for imitation learning. This makes data collection as simple as having a person perform the task while the robot mirrors their movements.

HumanPlus relies on two decoder-only transformer models that operate at different stages of the learning pipeline. The Humanoid Shadowing Transformer (HST) is the low-level controller that converts retargeted human joint trajectories into stable humanoid motion. It is trained in simulation via reinforcement learning (PPO) on the AMASS dataset, which contains 40 hours and over 11,000 unique sequences of human motion capture data. The HST takes as input the current humanoid joint state and the retargeted target pose from the human operator, and outputs joint-position commands that achieve the target pose while maintaining balance and physical feasibility. A key property of the HST is zero-shot sim-to-real transfer -- the policy trained in simulation deploys directly to the physical humanoid without any real-world fine-tuning.

The retargeting pipeline bridges the kinematic gap between human and humanoid bodies. Human 3D pose estimation (using models like WHAM) extracts body and hand joint positions from a single RGB camera at 30 Hz. These human joint positions are then mapped to the humanoid's joint space through kinematic retargeting that accounts for differences in limb lengths, joint limits, and morphology. The retargeted targets are physically achievable poses for the humanoid, not raw human joint angles.

The Human Imitation Transformer (HIT) is the high-level autonomous policy that learns task-specific skills from demonstration data collected via shadowing. During data collection, a human operator performs the task while the robot shadows their motion in real time, producing pairs of (egocentric observation, robot action) data. The HIT takes egocentric RGB images from the two head-mounted cameras as input and predicts joint-position action chunks using a Transformer architecture similar to ACT (Action Chunking with Transformers). It uses a CVAE (Conditional Variational Autoencoder) formulation to handle multi-modal action distributions.

The two-stage design (HST for motion transfer + HIT for task learning) creates a virtuous data collection loop. Because the HST enables real-time shadowing, data collection requires no specialized hardware beyond a camera pointed at the human operator. This makes it feasible to collect 10-40 demonstrations per task in minutes rather than hours, which is sufficient for the HIT to learn robust autonomous policies. The simplicity of the data collection pipeline is arguably HumanPlus's most important practical innovation.

HumanPlus has two distinct data requirements corresponding to its two transformer models. The Humanoid Shadowing Transformer (HST) is trained on human motion capture data in simulation. The published system uses the AMASS dataset, which aggregates motion capture data from multiple sources (CMU MoCap, Human3.6M, ACCAD, and others) into a unified SMPL format, totaling 40 hours and over 11,000 unique motion sequences. AMASS covers diverse activities including walking, running, dancing, sports, object manipulation, and daily activities. The HST is trained via PPO reinforcement learning in the MuJoCo physics simulator, where the humanoid attempts to track retargeted AMASS motions while maintaining balance. No real-world data is needed for this stage.

The Human Imitation Transformer (HIT) requires task-specific demonstration data collected on the physical humanoid via shadowing. For each target skill, a human operator performs the task while the robot mirrors their motion in real time. The resulting data consists of synchronized egocentric RGB frames (from the two head-mounted cameras, typically at 480x640 resolution) and 33-DoF joint-position recordings at 30 Hz. The published tasks required between 10 and 40 demonstrations each -- for example, 20 demonstrations for folding a sweatshirt, 40 for wearing a shoe and standing up, and 10 for greeting another robot.

For the HST, the critical data property is motion diversity. The 40 hours in AMASS cover a wide range of human activities, and this breadth is what enables the HST to track arbitrary human motion during deployment. If the target application involves motions significantly outside AMASS's coverage (e.g., highly specialized industrial movements or sports-specific techniques), supplementing with additional motion capture data improves shadowing fidelity. Each motion sequence should be in SMPL or SMPL-H format with 3D joint positions at 30+ Hz.

For the HIT, data quality is more important than quantity. Because only 10-40 demonstrations are used per task, each demonstration must be a clean, successful execution of the target skill. The egocentric camera viewpoint should be consistent across demonstrations, the objects should be in approximately similar starting configurations, and the human operator should execute the task smoothly without hesitation or errors. Failed or partial demonstrations degrade HIT performance significantly at these small sample sizes.

Claru provides data for both stages of the HumanPlus pipeline. For the HST training stage, we offer extensive human motion capture datasets in SMPL/SMPL-H format that can supplement or replace AMASS for broader motion coverage. Our motion data spans diverse activities including household manipulation, industrial assembly, warehouse operations, personal care, and fitness -- captured via high-fidelity optical motion capture systems (Vicon, OptiTrack) at 120 Hz and downsampled to the target training rate. This expanded motion vocabulary enables the HST to track a wider range of human motions during shadowing, particularly for manipulation-heavy tasks that are underrepresented in standard mocap datasets.

For the HIT training stage, Claru provides task-specific demonstration datasets collected on humanoid platforms via shadowing or direct teleoperation. Our collection pipeline captures synchronized egocentric RGB frames from head-mounted cameras at 30 Hz, full 33-DoF joint-position trajectories, and task success labels. We enforce strict data quality standards matching HumanPlus's requirements: each demonstration is a verified successful execution, camera viewpoints are consistent, and object configurations are controlled. For a typical HumanPlus skill, we deliver 20-50 high-quality demonstrations per task.

Beyond direct data provision, Claru's egocentric video catalog (3M+ clips of human daily activities captured from head-mounted cameras) provides a rich source of task-relevant visual data for pretraining or augmenting HIT's vision backbone. While HumanPlus does not currently use video pretraining, the architecture is compatible with frozen pretrained vision encoders, and teams exploring this extension can leverage our egocentric video for backbone pretraining. All data includes full provenance documentation, sensor calibration, and compatibility verification with HumanPlus's open-source codebase.