RGB Manipulation Video Dataset

RGB video of robot manipulation with monocular depth estimates provided as an annotation layer for training depth-aware grasping policies. Capture is RGB video — Claru does not capture hardware depth.

Dataset at a Glance

RGB
Capture modality
Derived
Monocular depth layer
Scoped
Volume per project
Custom
Annotation layers

Comparison with Public Datasets

How Claru's dataset compares to publicly available alternatives.

DatasetClipsHoursModalitiesEnvironmentsAnnotations
GraspNet-1Billion97K grasps~10RGB-D190 scenes (lab)6-DoF grasps
ACRONYM8.8M graspsN/A (sim)Synthetic depthSimulatedGrasp labels
Claru RGB Manipulation VideoScoped per projectScoped per projectRGB (depth as derived layer)Real manipulation setupsMonocular depth estimates, object masks, grasp/contact events, trajectories

Use Cases

6-DoF Grasp Planning

Predicting stable grasps from RGB video, with derived monocular depth as a geometry prior. Example models: GraspNet, Contact-GraspNet, AnyGrasp.

Depth-Conditioned Policies

Policies that take derived depth estimates as an extra input channel for spatial reasoning and collision avoidance. Example models: PerAct, 3D Diffusion Policy, RVT.

Object Pose Estimation

Estimating 6-DoF object poses from RGB video for manipulation planning. Example models: FoundationPose, MegaPose, BundleSDF.

Key References

  1. [1]Fang et al.. “GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping.” CVPR 2020, 2020. Link
  2. [2]Sundermeyer et al.. “Contact-GraspNet: Efficient 6-DoF Grasp Generation.” ICRA 2021, 2021. Link
  3. [3]Wen et al.. “FoundationPose: Unified 6D Pose Estimation and Tracking.” CVPR 2024, 2024. Link

How Claru Delivers This Data

Capture is RGB video. Monocular depth estimates may be provided as an annotation layer. Contact Claru to scope RGB capture and derived annotation requirements.

Frequently Asked Questions

No. Capture is RGB video — Claru does not capture registered depth from a depth sensor. Monocular depth estimates computed from the RGB frames may be provided as an annotation layer; they are relative estimates, not metric sensor depth.

Object sets are scoped per project and can span household items, tools, food, and industrial components in varying materials and sizes.

Annotators label grasp and contact events, grasp types, and hand-object interaction directly on the RGB video. There is no force/torque or depth sensor in the capture, so contact is labelled visually and any 3D localization relies on derived monocular depth estimates.

Request a Sample Pack

Get a curated sample of RGB manipulation video with derived depth and annotations to evaluate for your project.