CLARU
OpenMOSS-Team2025cc-by-nc-4.0

OmniAction

A large-scale multimodal dataset for proactive robot manipulation comprising 141,162 episodes across 112 skills and 748 objects, enriched with audio, visual, and contextual instruction data for cross-modal intention recognition.

Downloads42K
Episodes141162
Likes252

Why This Matters for Physical AI

This dataset advances physical AI by enabling robots to recognize human intentions from multimodal contextual cues (speech, sounds, visuals) rather than explicit instructions, bridging the gap between human-robot interaction in real-world collaborative scenarios.

Technical Profile

Modalities
rgbaudiolanguage
Environment
simulationlab
Task Types
manipulationpick_and_place
Episodes
141162
Data Format
RLDS
Annotation Types
language_instructionsaction_labels
License
cc-by-nc-4.0
Part of the OmniAction family

Community Signals

Top 5% by downloads

Access

View on HuggingFaceRead Paper

Need custom rgb data?

Claru builds purpose-built datasets for simulation applications with dense human annotations and quality assurance.

Request a Sample Pack

Related Datasets

OmniAction

A large-scale multimodal dataset for proactive robot manipulation comprising 141,162 episodes with cross-modal contextual instructions derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.

rgbaudiolanguage
45K downloads2025cc-by-nc-4.0

OmniAction

A large-scale multimodal dataset for proactive robot manipulation with 141,162 episodes covering contextual instruction following through spoken dialogue, environmental sounds, and visual cues.

rgbaudiolanguage
11K downloads2025cc-by-nc-4.0

OmniAction

A large-scale multimodal dataset for proactive robot manipulation comprising 141,162 episodes with contextual instructions derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.

rgbaudiolanguage
3K downloads2025cc-by-nc-4.0

Xperience-10M

A large-scale egocentric multimodal dataset of human experience containing 10 million interactions and 10,000 hours of synchronized first-person recordings with six video streams, audio, stereo depth, camera pose, hand mocap, full-body mocap, IMU, and hierarchical language annotations for embodied AI, robotics, and world modeling research.

rgbaudiodepthproprioception+3
2.3M downloadsother

Open-H-Embodiment

A multi-embodiment community-driven dataset of paired kinematics and video for training AI autonomy models in surgical robotics and ultrasound applications. The dataset includes tabletop exercises, clinical procedures, and simulations of healthcare robotics tasks.

rgbproprioception
118K downloads2026cc-by-4.0

Open-H-Embodiment

A community-driven, multi-embodiment dataset of paired kinematics and video for training and evaluating AI autonomy models in surgical robotics and ultrasound applications, including tabletop exercises, clinical procedures, and healthcare robotics simulations.

rgbproprioception
76K downloads2026cc-by-4.0