Anonymous-HEIR123cc-by-4.0

HEIR: Harness Egocentric Intent for Human–Robot Interactions

Name: HEIR: Harness Egocentric Intent for Human–Robot Interactions
Creator: Anonymous-HEIR123
License: https://creativecommons.org/licenses/by/4.0/
Keywords: egocentric_video, language, gaze, manipulation, navigation, human-robot-interaction, intent-grounding, home, work, kitchen

A collection of 300 processed session annotation JSON files from egocentric human-robot interaction scenarios, containing 1,836 episodes and 6,416 execution subtasks across navigation and manipulation tasks. The dataset provides hierarchical annotations of human intent, robot execution, and task completion grounded in egocentric video from home, work, and kitchen environments.

Downloads58
Episodes1836

Why This Matters for Physical AI

This dataset enables research on grounding human intent from egocentric perspectives for collaborative human-robot interaction, with hierarchical annotations linking natural language instructions to robot execution trajectories across real-world household and workplace tasks.

Technical Profile

Modalities: egocentric_videolanguagegaze
Environment: homeworkkitchen
Task Types: manipulationnavigationhuman-robot-interactionintent-grounding
Episodes: 1836
Data Format: JSON
Annotation Types: language_instructionsintent_labelsaction_labelstask_segmentationhierarchical_subtask_labels
License: cc-by-4.0

Part of the HEIR family

Access

View on HuggingFace Read Paper

Need custom egocentric_video data?

Claru builds purpose-built datasets for home applications with dense human annotations and quality assurance.

Request a Sample Pack

Related Datasets

Robo-MLT: Medical Laboratory Manipulation Datasets

Real-robot teleoperated demonstrations for two long-horizon medical laboratory workflows (blood gas analysis and tube sorting) collected on an Agilex dual-arm Static-Aloha platform with vision-language-action supervision.

rgbproprioceptionlanguage

null downloadsJul 2026apache-2.0gated

ACE-Data-0

A human-centric ambient capture dataset recording real home environments with synchronized multi-modal streams including egocentric video, exocentric video, human motion, object state, audio, and tactile sensing for embodied AI research.

rgbdepthimuaudio+3

245K downloadsOct 2026ace-data-0-research-license

In-The-Wild Humanoid Dataset

A large-scale dataset of whole-body humanoid robot learning in natural home environments, capturing human teleoperation demonstrations on Unitree G1 across real homes in Southeast Asia with varying layouts, lighting, and clutter.

rgbstereo_irproprioceptionimu+2

149K downloadsSep 2026

In-The-Wild Humanoid Dataset

A large-scale dataset of 500+ hours and 23K+ episodes of whole-body humanoid robot learning captured through human teleoperation demonstrations on Unitree G1 across real homes in Southeast Asia. The dataset includes synchronized visual observations, robot states, actions, and metadata for research on mobile manipulation, bimanual interaction, and long-horizon household skills.

rgbinfraredproprioceptionimu+2

149K downloadsSep 2026

OmniAction

A large-scale multimodal dataset for proactive robot manipulation comprising 141,162 episodes across 112 skills and 748 objects, enriched with audio, visual, and contextual instruction data for cross-modal intention recognition.

rgbaudiolanguage

117K downloadsMar 2026cc-by-nc-4.0

HiFi-UMI-2K: High-Fidelity Robot-Free Manipulation Data

A large-scale dataset of 2,000 hours of high-fidelity bimanual human manipulation demonstrations collected robot-free using the HiFi-UMI capture system, designed for action-grounded supervision for manipulation policy pre-training and deployment without requiring a robot during data collection.

rgbimuproprioceptionlanguage

114K downloadsJul 2026cc-by-4.0