Physical AI: Models That Understand and Act in the Real World

Physical AI is an umbrella term for artificial intelligence systems that model, understand, or interact with the physical world. It encompasses robot control policies that manipulate objects, world models that simulate physical dynamics, video generation systems that produce physically plausible footage, digital twins that mirror real-world processes, and perception systems that reconstruct 3D environments from sensor data.

The defining characteristic of physical AI is that it must respect the constraints of real-world physics. A language model can generate text about objects passing through walls; a physical AI system must understand that solid objects cannot interpenetrate. A language model operates in a discrete token space; physical AI operates in continuous spaces where small perturbations can have large consequences (dropping a glass versus placing it gently). This requirement for physical consistency makes physical AI fundamentally harder than digital AI and creates specific data requirements.

Physical AI models are trained on data that captures real-world physics at the resolution needed for the target application. Robot manipulation policies need synchronized camera images and action labels at 10-50 Hz. World models need diverse video showing how objects, materials, and environments change over time. Perception systems need 3D reconstructions, depth maps, and semantic labels. The common thread is that all physical AI training data must be grounded in reality — synthetic data can supplement but not replace real-world observations because simulators cannot perfectly model the complexity of physical interactions.

The commercial opportunity for physical AI is immense. McKinsey estimates the global market for physical AI-enabled systems (including robotics, autonomous vehicles, and industrial automation) will exceed $200 billion annually by 2030. The bottleneck is not algorithms — the Transformer architecture that powers language models also powers VLA models for robots. The bottleneck is training data: collecting, annotating, and curating the real-world data that physical AI systems need to learn from.

For teams building physical AI systems, the data strategy is the most important early decision. The choice of data modalities, collection scale, environment diversity, and annotation layers determines what the model can and cannot learn.

Physical AI data falls into three categories by collection difficulty and cost. Passive video (recording scenes without robot interaction) is the cheapest to collect at scale — any camera can capture it. But passive video provides observation data without action labels, useful for pretraining vision encoders and world models but insufficient for learning robot control. Human demonstration data (egocentric video of people performing tasks) provides observation and implicit action information, useful for pretraining visuomotor representations at moderate cost. Robot teleoperation data (observations paired with explicit robot actions) is the most expensive but provides the exact training signal that manipulation policies need.

The practical approach that most teams adopt is a data pyramid. At the base, large volumes (10,000+ hours) of passive and human demonstration video pretrain the visual backbone. In the middle, moderate volumes (1,000-10,000 episodes) of cross-embodiment robot data pretrain the policy backbone. At the top, smaller volumes (1,000-5,000 episodes) of on-hardware teleoperation data fine-tune for the specific deployment scenario.

Environment diversity is often underestimated. Physical AI systems that train in a single lab environment achieve impressive demo results but fail when deployed in a different setting with different lighting, surfaces, and object arrangements. Production systems need training data from dozens to hundreds of distinct environments. This is fundamentally a logistics problem: data collection must be geographically distributed, which requires a global network of collection sites and trained operators.

Claru is a physical AI data company. We provide the three layers of the training data pyramid: large-scale egocentric video for visual pretraining, cross-environment human demonstration data for representation learning, and on-hardware teleoperation data for policy fine-tuning.

Our positioning within the physical AI ecosystem is as the data infrastructure layer. While model companies (NVIDIA, Physical Intelligence, Figure AI) build architectures and hardware, and simulation companies build virtual environments, Claru provides the real-world data that these systems cannot learn without. With 386,000+ annotated clips, 10,000+ trained collectors, and coverage across 100+ cities and 12+ environment types, Claru delivers physical AI training data at production scale.

Physical AI is an umbrella term for artificial intelligence systems that model, understand, or interact with the physical world. It encompasses robot control policies that manipulate objects, world models that simulate physical dynamics, video generation systems that produce physically plausible footage, digital twins that mirror real-world processes, and perception systems that reconstruct 3D environments from sensor data. The term gained prominence when NVIDIA CEO Jensen Huang used it during his GTC 2024 keynote to describe the next frontier of AI after language models.

The defining characteristic of physical AI is that it must respect the constraints of real-world physics. A language model can generate text about objects passing through walls; a physical AI system must understand that solid objects cannot interpenetrate. A language model operates in a discrete token space; physical AI operates in continuous spaces where small perturbations can have large consequences — dropping a glass versus placing it gently on a table. This requirement for physical consistency makes physical AI fundamentally harder than digital AI and creates specific data requirements that text and image datasets cannot satisfy.

The commercial opportunity is enormous. McKinsey estimates the global market for physical AI-enabled systems — including robotics, autonomous vehicles, and industrial automation — will exceed $200 billion annually by 2030. The bottleneck is not algorithms. The Transformer architecture that powers language models also powers VLA models for robots. The bottleneck is training data: collecting, annotating, and curating the real-world data that physical AI systems need to learn from.

Physical AI data falls into three layers organized by collection difficulty, cost, and training value. At the base of the pyramid: passive video — recording scenes without robot interaction — is the cheapest to collect at scale. Any camera can capture it. Passive video provides observation data without action labels, useful for pretraining vision encoders and world models but insufficient for learning robot control directly. Internet video, egocentric recordings, and surveillance footage all fall in this category.

In the middle of the pyramid: human demonstration data — egocentric video of people performing tasks — provides observation and implicit action information. When a human picks up a cup on camera, the video captures the visual trajectory of the hand, the object, and the scene change. This data is more expensive to collect than passive video (you need a camera-wearing person performing specific tasks) but less expensive than robot data. It is particularly valuable for pretraining visuomotor representations at moderate cost.

At the top: robot teleoperation data — synchronized observations paired with explicit robot actions — is the most expensive but provides the exact training signal that manipulation policies need. Each demonstration requires a physical robot, a skilled teleoperator, and the target objects and environment. This data directly trains the action prediction models that control robots. The practical approach that most teams adopt is to build all three layers: large volumes of passive video at the base, moderate volumes of human demonstration data in the middle, and targeted robot teleoperation data at the top.

NVIDIA has positioned itself as the infrastructure layer for physical AI through three platforms: Isaac Sim for physics simulation, Cosmos for world foundation models trained on physical interaction video, and GR00T N1 for humanoid robot control. Jensen Huang's framing of physical AI as a product category catalyzed industry investment and established NVIDIA's GPU and simulation platforms as the default development environment for the field.

Google DeepMind's robotics research established the VLA (Vision-Language-Action) paradigm with RT-2, demonstrating that large vision-language models can be co-trained on web data and robot demonstrations to produce models with emergent manipulation reasoning. DeepMind's partnership with Boston Dynamics extends this approach to the most mechanically capable humanoid platform. Physical Intelligence (founded by former Google researchers) built pi-zero, a VLA with flow-matching action prediction that targets production-grade manipulation.

On the humanoid side, Figure AI ($2.6B funded, OpenAI partnership), 1X Technologies (data-centric approach, $20K consumer humanoid), Agility Robotics (Digit in Amazon warehouses), and Tesla (Optimus) are all building physical AI systems that require massive training data investments. The common thread across all these companies is that model architecture is largely converged (Transformers + diffusion/flow for actions) and the competitive differentiator is training data quality, diversity, and scale.

The sim-to-real gap is the fundamental data challenge for physical AI. Simulated environments — even state-of-the-art platforms like NVIDIA Isaac Sim and MuJoCo — cannot perfectly model the complexity of real-world physical interactions. Contact dynamics between a robot fingertip and a real object depend on surface roughness, material compliance, moisture, temperature, and wear patterns that no simulator captures at sufficient fidelity. Deformable objects (fabric, food, paper) are especially difficult to simulate because their behavior depends on material properties that vary across instances.

NVIDIA and other simulation providers invest in improving simulation fidelity, but the gap is fundamental rather than engineering. Real-world physics is governed by partial differential equations with parameters that vary continuously across materials and conditions. Simulation discretizes these equations on a finite grid with approximate material parameters. The approximation error propagates through learned policies, causing failures when the policy encounters real-world conditions that fall outside the simulator's approximation envelope.

This is why all leading physical AI companies — from NVIDIA to Physical Intelligence to Figure AI — invest in real-world data collection alongside simulation. The industry consensus is that simulation reduces the amount of real-world data needed but does not eliminate the need entirely. The most capable physical AI systems use a combination: simulation for pretraining and policy search, real-world data for fine-tuning and validation. Claru exists to provide the real-world layer of this data stack at production scale.