Monocular Depth Estimation: Predicting 3D Depth from a Single Camera

Monocular depth estimation is the task of predicting the distance from the camera to every visible surface in a scene, using only a single RGB image as input. The output is a dense depth map — a 2D array where each pixel value represents the estimated depth (distance from the camera plane) at that spatial location. This is an inherently ill-posed problem because infinitely many 3D scenes can project to the same 2D image, requiring the model to leverage learned priors about scene geometry, object sizes, texture gradients, and perspective cues.

MDE models are categorized by their output type. Relative depth models predict ordinal depth ordering without metric scale — they can determine that surface A is closer than surface B, but not by how many centimeters. Metric depth models predict absolute distances in real-world units, typically meters. A third hybrid category produces scale-invariant depth that can be converted to metric depth using a single known reference (such as the camera's focal length or a known object dimension).

Architecturally, modern MDE models use encoder-decoder designs with vision transformer (ViT) backbones. The encoder processes the input image through a pretrained ViT to extract multi-scale features that encode both local texture and global scene structure. The decoder upsamples these features back to the input resolution, predicting depth at every pixel. DPT (Ranftl et al., 2021) established this ViT-based architecture, and subsequent models like MiDaS v3.1, ZoeDepth, Depth Anything, and Metric3D refine the encoder-decoder design with better training data, loss functions, and domain adaptation strategies.

For robotics, MDE converts any RGB camera into a pseudo-depth sensor. This is valuable because depth sensors (LiDAR, structured light, time-of-flight) are expensive, have limited range or resolution, fail on certain surfaces (transparent, highly reflective), and add size and weight to the robot platform. MDE from a standard camera provides dense depth at any resolution, works on all surfaces, and adds zero hardware cost. The tradeoff is reduced accuracy compared to active depth sensors, particularly for metric depth where absolute distance matters for manipulation and collision avoidance.

Deploying MDE in a robotics system requires decisions about model selection, metric calibration, and integration with downstream perception modules.

Model selection depends on the accuracy-latency tradeoff. Depth Anything V2 with a ViT-Large backbone achieves state-of-the-art accuracy but runs at 15-30 FPS on an NVIDIA Jetson AGX Orin. The ViT-Small variant runs at 60+ FPS with slightly reduced accuracy, suitable for real-time navigation. For offline processing (dataset enrichment, 3D reconstruction), larger models and multi-frame fusion methods provide the highest quality. Teams should benchmark candidate models on images from their deployment environment before committing to an architecture.

Metric calibration is necessary for any application requiring absolute distances. Relative depth models must be aligned to metric scale, typically by capturing a small calibration dataset (50-200 frames) with a known depth sensor and fitting a scale-and-shift transformation. ZoeDepth and Metric3D avoid this step by directly predicting metric depth, but their accuracy degrades on environments that differ significantly from their training distribution. For best results, Claru recommends fine-tuning metric heads on 500-2,000 frames from the target environment captured with a calibrated depth sensor.

Integration with downstream modules determines the practical value of MDE predictions. For obstacle avoidance, depth maps are thresholded to create binary free-space maps. For 3D reconstruction, per-pixel depth is unprojected to point clouds using camera intrinsics. For grasp planning, depth maps are fused with instance segmentation masks to estimate per-object 3D geometry. Each integration pathway has different accuracy requirements — navigation tolerates 10% error, while grasp planning requires sub-centimeter accuracy that typically demands sensor fusion (combining MDE with a depth camera or stereo pair) rather than MDE alone.

Claru captures the paired RGB-depth training data that monocular depth estimation models need for domain-specific fine-tuning and metric calibration. Our data collection network uses calibrated Intel RealSense and Azure Kinect sensors across 100+ cities, producing synchronized RGB-depth frames in the diverse real-world environments — homes, warehouses, retail floors, outdoor spaces — where robots actually operate. Each frame includes camera intrinsics, depth sensor specifications, and environment metadata. For teams deploying Depth Anything or ZoeDepth, Claru provides the 500-5,000 frame calibration datasets that convert general-purpose MDE into metric depth with deployment-specific accuracy. Our enrichment pipeline also runs Depth Anything V2 on existing RGB-only video in our 386,000+ clip catalog, producing pseudo-depth annotations that extend dataset utility for teams whose downstream models consume depth features.