Spatial Intelligence: How AI Systems Learn to Understand Physical Space

Spatial intelligence is the capacity of an AI system to perceive, internally represent, reason about, and physically act within three-dimensional environments. Where traditional computer vision operates on flat images — classifying objects, detecting faces, reading text — spatial intelligence requires understanding the full 3D structure of a scene: the geometry of surfaces, the volumetric extent of objects, the traversability of paths, the reachability of targets, and the physical affordances of every element in the environment.

The concept spans several interconnected capabilities. 3D scene understanding reconstructs the geometry and semantics of an environment from sensor input — building an internal model of what is where, how large it is, and what it is made of. Object affordance recognition determines what actions are possible given the physical properties of objects: a handle affords grasping, a flat surface affords placement, a gap affords passage. Spatial relationship reasoning encodes the relative positions, orientations, and distances between entities — understanding that object A is behind object B, above object C, and within reaching distance of the robot's end effector. Navigation uses this spatial model to plan collision-free paths through environments. Manipulation uses it to plan contact-rich interactions with objects — grasping, pushing, inserting, pouring — that respect physical constraints like gravity, friction, and deformability.

Spatial intelligence is distinct from spatial computing, the term Apple popularized with Vision Pro to describe augmented and mixed reality interaction paradigms. Spatial computing is a human-computer interface concept; spatial intelligence is a machine capability. The two converge when AR/MR systems use AI to understand room geometry, but spatial intelligence extends far beyond consumer devices into industrial robotics, autonomous vehicles, construction automation, and any domain where machines must reason about physical space.

The critical data insight is that spatial intelligence cannot be learned from internet-scraped 2D images. Models need training data captured in physical environments with calibrated sensors that record depth, multi-view geometry, and egocentric perspective: stereo and RGB-D video, LiDAR point clouds, IMU traces, and GPS coordinates. This is fundamentally different from the web-crawled datasets that powered the language and 2D vision revolutions, and it is why companies building spatial AI face a data bottleneck that cannot be solved by scaling existing internet datasets.

For teams building spatial AI systems, the training data pipeline is fundamentally different from the text and 2D image pipelines that powered the previous generation of AI. Understanding these differences is essential to avoiding months of wasted effort collecting the wrong data.

First, spatial intelligence models need multi-view captures, not single photographs. A single image of a room tells a model what the room looks like from one angle; multiple images from different viewpoints — or continuous video as a camera moves through the space — provide the geometric constraints the model needs to reconstruct 3D structure. Capture protocols must ensure sufficient viewpoint diversity: baseline distances between camera positions, coverage of all surfaces and objects, and overlap between frames for feature matching. Claru's collection methodology uses structured capture patterns — systematic walkthroughs with calibrated head-mounted cameras — that guarantee the multi-view coverage reconstruction algorithms require.

Second, depth data is not optional. While monocular depth estimation has improved dramatically (Depth Anything V2, Metric3D), these models are themselves trained on ground-truth depth from structured-light sensors, stereo cameras, and LiDAR. For training spatial intelligence models, ground-truth depth from calibrated sensors provides supervision that is orders of magnitude more reliable than predicted depth. Claru's wearable capture devices include synchronized depth sensors alongside RGB cameras, producing paired RGB-D data at every frame.

Third, egocentric perspective matters. Most internet images are taken from a photographer's compositional viewpoint — centered, well-framed, at eye level. But a robot or embodied agent sees the world from its own body-centric perspective: looking down at a countertop from above, reaching toward objects at odd angles, navigating tight spaces with partial occlusion. Egocentric video from head-mounted cameras naturally captures this perspective, including the hand-object interactions that manipulation policies need to learn from. Claru's collector network captures hundreds of hours of egocentric video monthly across diverse environments, providing the embodied viewpoint that spatial AI models require.

Fourth, metadata transforms raw captures into spatial data. GPS coordinates ground captures in global position. IMU (inertial measurement unit) data provides orientation and acceleration for motion estimation. Camera intrinsic and extrinsic calibration parameters enable metric-scale 3D reconstruction. Without this metadata, video is just pixels; with it, video becomes a spatially grounded record of a physical environment. Claru's capture pipeline records and validates all sensor metadata in real time, ensuring that every frame can be precisely located in 3D space.

Finally, spatial annotations add semantic meaning to geometric data. Room layout estimation, object bounding volumes in 3D, surface material labels, affordance annotations (graspable, sittable, openable), and spatial relationship graphs (object A is on top of object B, 30cm to the left of object C) turn raw sensor data into the structured supervision spatial intelligence models learn from. These annotations require human judgment combined with 3D visualization tools — a fundamentally different workflow from 2D image labeling.

Spatial intelligence sits at the exact intersection of Claru's core capabilities. Our collection network — 10,000+ collectors equipped with wearable cameras and depth sensors operating across 100+ cities — is purpose-built to produce the real-world spatial data that 3D foundation models, embodied navigation systems, and manipulation planners require. Every Claru capture session records synchronized RGB video, depth streams, and IMU/GPS metadata from egocentric perspectives in diverse physical environments, directly addressing the data bottleneck that spatial AI teams face. Where internet-scraped 2D datasets provide appearance without geometry, and synthetic environments provide geometry without visual realism, Claru's physically captured datasets provide both: real visual appearance grounded in real 3D structure, with the sensor metadata needed for metric-scale reconstruction. Our annotation pipeline adds spatial semantics — room layouts, object affordances, 3D bounding volumes, surface materials, and spatial relationship graphs — transforming raw sensor captures into structured training data for spatial intelligence. For teams building the next generation of spatially intelligent systems, Claru eliminates the most expensive and time-consuming step: acquiring diverse, high-quality, spatially grounded real-world data.