Depth Data: 3D Perception for Robot Manipulation and Navigation

Depth data is a spatial measurement modality that encodes the distance from a camera sensor to each visible surface point in the scene. Represented as a 2D image where pixel values indicate distance (typically in millimeters or meters), a depth map transforms the flat RGB image into a 2.5D representation that captures the three-dimensional geometry of the environment. When combined with camera intrinsic parameters (focal length and principal point), each depth pixel can be un-projected into a 3D point in camera coordinates, enabling construction of dense point clouds from a single viewpoint.

In robotics, depth data serves as the primary input for spatial reasoning. Grasp planning algorithms like GraspNet and Contact-GraspNet operate on point clouds derived from depth maps to compute 6-DOF grasp poses. Navigation policies use depth to estimate obstacle distances and traversability. Manipulation policies use depth to determine object heights, distances, and surface orientations that RGB images encode only implicitly through learned visual features.

The relationship between depth data and RGB data is complementary. RGB provides appearance, texture, color, and semantic information (what is this object?). Depth provides geometry, distance, and 3D structure (where is this object and how big is it?). RGB-D — the combination of aligned RGB and depth images — has become the standard observation format for indoor robotics. Most robot learning frameworks (including those based on Open X-Embodiment) accept RGB-D as a primary input modality, with the depth channel processed either as a fourth image channel or converted to a point cloud and processed by 3D encoders like PointNet++ or sparse 3D convolutions.

Modern monocular depth estimation models (Depth Anything V2, Metric3D, UniDepth) can generate depth maps from single RGB images, eliminating the need for dedicated depth sensors at inference time. However, these predictions are trained on datasets of real depth sensor data, creating a data dependency: the quality of monocular depth models depends directly on the quality and diversity of the sensor-based depth data used for training and evaluation.

For teams building robot learning datasets with depth, sensor selection and mounting are the first decisions. For tabletop manipulation with a single arm, the standard setup is one RealSense D435 mounted overhead (0.5-1.5m from the workspace) and optionally one D405 mounted on the wrist (0.1-0.5m from objects). The overhead camera provides scene context; the wrist camera provides close-range precision during grasping. Both must be calibrated (intrinsics and extrinsics) and time-synchronized with the robot's control system.

Depth data quality control is essential because raw sensor output contains artifacts. A standard preprocessing pipeline includes: (1) depth filtering to remove readings outside the sensor's reliable range (e.g., clipping below 0.3m and above 3m for the D435), (2) hole filling via inpainting or depth completion for pixels with missing readings (typically 5-15% of pixels on scenes with reflective objects), (3) temporal smoothing over 3-5 frames for static scenes, and (4) optional bilateral filtering to smooth noise while preserving edges. This pipeline should be applied before the data enters the training dataset.

For large-scale data collection where deploying depth sensors at every capture site is impractical, monocular depth estimation provides a scalable alternative. Depth Anything V2 runs at 30+ fps on a consumer GPU and produces relative depth maps from RGB video. For robotics applications requiring metric depth, Metric3D or ZoeDepth add absolute scale estimation, achieving 5-15% relative error at manipulation distances. The practical workflow is to collect RGB-only video with Claru's distributed collector network, then run monocular depth estimation as a batch processing step to add depth channels. This hybrid approach trades some depth accuracy for massive scale — thousands of diverse environments that would be infeasible to instrument with depth sensors.

Point cloud generation from depth maps requires careful implementation. The standard un-projection formula (X = (u-cx)*Z/fx, Y = (v-cy)*Z/fy) assumes a pinhole camera model. Real lenses have distortion (radial and tangential) that must be corrected before un-projection, using the distortion coefficients from camera calibration. Failing to correct for distortion produces point clouds with systematic geometric errors at the image periphery, which degrades 3D policy performance.

Claru provides RGB-D datasets with hardware depth from calibrated RealSense sensors, as well as monocular depth augmentation for our large-scale RGB-only video collections. Every depth-equipped capture includes factory-calibrated intrinsics, per-frame extrinsics, and a preprocessed depth channel with hole filling and temporal smoothing applied. Point cloud exports in PLY or PCD format are available on request.

For teams that need depth data at scale across diverse environments, Claru's monocular depth pipeline runs Depth Anything V2 on our 386,000+ clip catalog, adding depth channels to egocentric and manipulation videos captured in 100+ cities. This provides the environmental diversity needed for robust 3D policy training without requiring depth sensors at every capture location. Datasets ship with both raw sensor depth (where available) and model-estimated depth, letting teams choose the tradeoff between accuracy and scale that fits their application.