Training Data for UC Berkeley AUTOLAB
AUTOLAB pioneered neural grasp planning with Dex-Net. Here is how diverse real-world grasping data bridges the gap between simulated and deployed performance.
About UC Berkeley AUTOLAB
AUTOLAB at UC Berkeley, led by Professor Ken Goldberg, is a leading robotics research lab focused on automating grasping, manipulation, and surgical robotics. The lab has produced foundational work on Dex-Net (neural grasp planning networks), FogROS (cloud robotics infrastructure), and large-scale grasp planning datasets. AUTOLAB supports 30+ postdocs, PhD students, and undergrads pursuing projects in robust robot grasping for warehouses, homes, and surgery.
AUTOLAB at a Glance
Known Data Requirements
AUTOLAB's grasping research requires massive real-world grasp attempt datasets to train and validate neural grasp planners. Dex-Net models achieve 95%+ success in lab conditions but face a persistent sim-to-real gap in deployment. Their cloud robotics work (FogROS) enables distributed data collection but requires diverse environment data to demonstrate value. Expanding beyond standard object sets to real-world product diversity is the critical frontier.
Real-world grasp attempt data at scale
Source: Dex-Net series (Mahler et al., RSS 2017 through 2023) and grasping benchmark research
Thousands of real-world grasp attempts on diverse objects with success/failure labels, approach angles, and grasp quality metrics for training and validating neural grasp planners beyond laboratory object sets.
Bin picking data with cluttered scenes
Source: AUTOLAB industrial bin picking research and logistics partnerships
Multi-object bin picking recordings showing grasp selection, execution, and replanning in cluttered bins with objects of varying geometries, materials, and weights — reflecting real warehouse product assortments.
Diverse object datasets for grasp generalization
Source: Dex-Net training data requirements for novel object grasping
RGB-D scans and manipulation recordings of thousands of real-world objects spanning household, industrial, and commercial categories for training grasps that generalize to unseen objects beyond YCB and Google Scanned Objects.
Surgical manipulation data
Source: AUTOLAB surgical robotics research and da Vinci platform work
Fine-grained manipulation recordings in surgical contexts — suturing, tissue manipulation, instrument handling — with sub-millimeter precision tracking for surgical robot learning.
Multi-site distributed grasp data
Source: FogROS 2 cloud robotics platform (Ichnowski et al., ICRA 2023)
Grasp attempt data collected simultaneously from robots at multiple physical locations via cloud infrastructure, capturing environmental diversity that single-site collection cannot achieve.
How Claru Data Addresses These Needs
| Lab Need | Claru Offering | Rationale |
|---|---|---|
| Real-world grasp attempt data at scale | Custom Grasping Data Collection | Claru can coordinate grasp attempt data collection across multiple sites with standardized protocols, producing the scale and diversity of real-world grasp data that single-lab collection cannot achieve. Each site contributes unique objects and environmental conditions. |
| Bin picking data with cluttered scenes | Manipulation Trajectory Dataset + Custom Collection | Claru's manipulation data includes cluttered scene interactions. Targeted collection campaigns can produce bin-picking-specific data with varied object assortments and bin configurations reflecting real warehouse product distributions. |
| Diverse object datasets for grasp generalization | Egocentric Activity Dataset + Custom Object Scanning | Claru's egocentric data captures objects in natural environments across 100+ cities. Custom scanning campaigns can produce RGB-D object scans from globally diverse product categories — household items, tools, food packaging, electronics — far beyond standard academic object sets. |
| Multi-site distributed grasp data | Distributed Multi-Site Collection Campaigns | Claru's global collector network mirrors the distributed collection model that FogROS enables technically. By coordinating standardized grasp experiments across many physical locations simultaneously, Claru provides the environmental diversity that validates FogROS's distributed architecture. |
Technical Data Analysis
UC Berkeley's AUTOLAB has been at the forefront of data-driven grasping for over a decade. The Dex-Net series demonstrated that neural networks trained on large synthetic grasp datasets can achieve robust bin picking — Dex-Net 2.0 used 6.7 million synthetic data points to train a robot that could pick up and move real objects with a 99% success rate in controlled laboratory conditions. Dex-Net 4.0 extended this to ambidextrous grasping with both suction and parallel-jaw grippers. These results established that data-driven grasping can match or exceed hand-engineered grasp planners.
However, the persistent gap between synthetic grasp quality predictions and real-world grasp success rates reveals the limits of simulation-only training. Dex-Net's synthetic data is generated by rendering 3D object models and computing analytic grasp metrics — but real objects have material properties (surface texture, compliance, friction, contamination), geometric features (rounded edges, thin handles, deformable packaging), and environmental conditions (lighting, clutter, bin geometry) that simulation models imprecisely. Each of these factors contributes to grasp failures that synthetic training cannot anticipate.
AUTOLAB's FogROS cloud robotics platform creates an interesting data collection paradigm. FogROS enables robots in different physical locations to share computation and data through cloud infrastructure. If robots at multiple sites are collecting grasp data simultaneously, the resulting dataset captures far more environmental and object diversity than any single-site collection campaign. Claru's distributed collection model mirrors this philosophy at a larger geographic scale — coordinating standardized grasp experiments across dozens of locations to produce datasets with the diversity that makes neural grasp planners robust in deployment.
The bin picking research has direct commercial implications that make data quality critical. Amazon, logistics companies, and manufacturers all need robust bin picking systems for warehouse automation. Training these systems requires data from real bins with authentic product assortments — not the standardized object sets (YCB, Google Scanned Objects) used in laboratory evaluations. Real warehouse products include unusual geometries (L-shaped brackets, nested cups), challenging materials (transparent packaging, reflective surfaces), and extreme size variation (tiny screws to large boxes) that academic object sets do not adequately represent.
The surgical robotics dimension adds requirements for precision manipulation data at a different scale. AUTOLAB's work on surgical robot learning requires demonstrations of tasks like suturing, tissue retraction, and instrument exchange with sub-millimeter tracking accuracy. While distinct from warehouse grasping, the underlying principle is the same: real-world data collected under authentic conditions produces better policies than simulation or laboratory approximations.
Key Research & References
- [1]Mahler et al.. “Dex-Net 2.0: Deep Learning to Plan Robust Grasps.” RSS 2017, 2017. Link
- [2]Ichnowski et al.. “FogROS 2: An Adaptive and Extensible Platform for Cloud and Fog Robotics.” ICRA 2023, 2023. Link
- [3]Satish et al.. “On-Policy Dataset Synthesis for Learning Robot Grasping Policies Using Fully Convolutional Deep Networks.” RA-L 2019, 2019. Link
- [4]Mahler et al.. “Learning Ambidextrous Robot Grasping Policies.” Science Robotics, Vol 4, 2019. Link
- [5]Danielczuk et al.. “Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data.” ICRA 2019, 2019. Link
- [6]Goldberg, K. et al.. “Beyond Planar Grasping: Modeling and Planning for Multi-Fingered Manipulation.” AUTOLAB Technical Report, 2024. Link
Frequently Asked Questions
Dex-Net trains on millions of simulated grasps computed over 3D object meshes but faces a persistent sim-to-real gap. Real objects have material properties, surface conditions, friction variations, and geometric details that simulation misses. Real-world grasp attempt data with success/failure labels helps calibrate and validate neural grasp planners for authentic deployment conditions.
Lab data uses standardized object sets (YCB with 77 objects, Google Scanned Objects with ~1,000) in controlled conditions with consistent lighting and clean surfaces. Deployment data features real products from warehouses with unexpected geometries, materials, contamination, and clutter. Policies trained only on lab data fail on the long tail of real-world objects that warehouses handle.
AUTOLAB's FogROS 2 platform allows robots at multiple sites to share computation and data through cloud infrastructure. Multiple robots collecting grasp data simultaneously from different locations naturally produce more diverse datasets than single-site collection. Claru's distributed collector network extends this principle further — coordinating data collection across 100+ locations.
Dex-Net 4.0 combines parallel-jaw and suction grasping into a single ambidextrous system that automatically selects the best gripper type for each object and grasp pose. Earlier versions handled only one gripper type. This ambidextrous approach requires training data from both gripper types across the same object diversity — doubling the data requirement compared to single-gripper systems.
Surgical robotics applies the same data-driven manipulation principles as warehouse grasping but at much higher precision. Tasks like suturing require sub-millimeter accuracy, tissue manipulation demands force-aware control, and instrument exchange needs reliable grasping under surgical conditions. The underlying approach — learning from demonstration data — is shared, but surgical data collection requires specialized instruments and environments.
Real-World Data for Grasp Planning
Discuss diverse grasping and manipulation data for AUTOLAB's neural grasp planning research.