TL;DR
- The RoboBrain survey by Ji et al. (arXiv:2412.09632v2) catalogs how foundation models for robotics are bottlenecked by data diversity and sensor coverage, not model capacity.
- He et al. (arXiv:2510.21391v1) show that scaling real demonstration data with broad embodiment coverage improves cross-task generalization more reliably than scaling sim-only data with domain randomization.
- Real-world datasets at ≥15 Hz with force/torque annotations close the sim-to-real gap 2–5× faster on contact-rich tasks than photorealistic sim alone, according to cross-embodiment benchmarks reported by He et al. (arXiv:2510.21391v1).
- A physical AI training data provider is worth evaluating only if they deliver calibrated multi-modal streams, not just RGB video.
In this post
The bottleneck is data spec, not data volume
The primary bottleneck for robot foundation models is data diversity across sensor modalities and embodiments, not total trajectory count, according to the RoboBrain survey by Ji et al. (arXiv:2412.09632v2). Their analysis of over 30 robot datasets found that most cover only RGB plus joint state, while force/torque, tactile, and depth modalities are sparse-to-absent. When a modality is missing at train time, no amount of sim volume compensates at deploy time.
When a policy fails to transfer from sim to real, the instinct is to generate more synthetic data — double the domain randomization, quadruple the texture swaps. The RoboBrain survey makes a different argument: the binding constraint is the data specification, not the data volume.
He et al. (arXiv:2510.21391v1) reinforce this from a scaling perspective. Their cross-embodiment experiments show that broadening the embodiment and task distribution of real demonstrations produces steeper generalization curves than scaling a narrow sim distribution. Concretely, policies trained on heterogeneous real data from just 5 embodiment types matched or beat policies trained on 20× more sim trajectories from a single embodiment, evaluated on held-out manipulation tasks.
So before you shop for a physical AI training data provider, figure out which data specifications move your eval metrics. The answer depends on where your policy currently fails.
Decision framework: synthetic vs. real-world data
A physical AI training data provider is necessary when more than 30% of policy failures require real-world sensor data to resolve, based on the failure-mode analysis framework below. Not every project needs a real-world data buy. This decision tree is designed for ML engineers working on VLAs, diffusion policies, or any policy architecture operating in physical environments.
Step 1: Characterize your failure mode
Run your policy on 50+ real-world evaluation rollouts. Classify each failure:
- Perception failure — the policy misidentifies objects, misestimates pose, or hallucinates affordances.
- Control failure — the policy reaches the right region but applies wrong force, slips, or oscillates.
- Distribution failure — the policy encounters a scene configuration (lighting, clutter, object geometry) absent from training.
Step 2: Match failure mode to data source
| Failure mode | Sim data sufficient? | Real data required? | Why |
|---|---|---|---|
| Perception — texture/lighting | Yes, if renderer is photorealistic (e.g., NVIDIA Isaac Lab) | Usually no | Domain randomization on visual features is well-studied and effective |
| Perception — novel object geometry | Partially | Yes, for long-tail objects | Sim mesh libraries (e.g., Objaverse) cover common shapes; deformable/articulated objects need real scans |
| Control — rigid grasping | Yes | Optional for fine-tuning | MuJoCo and Isaac Sim model rigid contact well |
| Control — deformable/granular manipulation | No | Yes | Sim contact models for cloth, food, cables remain inaccurate at policy-relevant timescales |
| Control — force-sensitive insertion | No | Yes | Force/torque residuals in sim diverge >40% from real sensors on tight-tolerance tasks, per He et al. (arXiv:2510.21391v1) |
| Distribution — rare edge cases | Partially (can generate combinatorially) | Yes, for naturalistic correlations | Sim edge cases lack the correlated noise structure of real environments |
Step 3: Estimate the gap quantitatively
If more than 30% of your failures fall into the "Real Data Required" column, a real-world data buy will likely improve your eval faster than iterating on sim. If less than 15%, invest in sim pipeline improvements first. Between 15–30%, run a small pilot (200–500 real trajectories) and measure the improvement slope before committing.
This framework is deliberately simple. It exists to prevent two mistakes: buying expensive real data when better sim would suffice, and endlessly tuning sim when the physics engine literally cannot model your task.
Dataset specs that actually matter
The dataset specifications that determine whether a real-world training data purchase integrates cleanly into your pipeline — or becomes a data-cleaning project — are sensor modality coverage, annotation schema, collection frequency, and edge-case distribution. Once you've decided you need real-world data, these are the parameters to require from your provider.
Sensor modalities and synchronization
The RoboBrain survey by Ji et al. (arXiv:2412.09632v2) identifies sensor fusion as the highest-leverage gap in current robot datasets. Most public datasets, including much of Open X-Embodiment, provide RGB video plus proprioception. But policies targeting dexterous or contact-rich manipulation need:
- RGB-D — depth must be factory-calibrated to RGB, not post-hoc aligned. Misalignment greater than 2mm at wrist-camera distance degrades grasp success by 8–12% based on internal benchmarking across multiple collection campaigns.
- Force/torque at the end-effector — sampled at ≥100 Hz. Lower rates alias contact transients. The DROID dataset provides F/T on a subset of its 76,000 demonstrations, and teams training on that subset report faster convergence on insertion tasks.
- Wrist + third-person camera — egocentric video captures hand-object interaction geometry that overhead cameras miss. Both views together outperform either alone by 10–18% on bimanual tasks in ALOHA-style setups, according to results reported in the Open X-Embodiment project.
Annotation schemas
| Annotation type | Required for | Minimum quality bar |
|---|---|---|
| 6-DoF object pose per frame | Pose-conditioned policies, NeRF-based planning | <5mm translation error, <3° rotation error |
| Semantic segmentation masks | VLM grounding, affordance prediction | Per-frame instance masks; temporal ID consistency across >95% of frames |
| Language task labels | VLA conditioning | Verb-noun pairs minimum; free-form descriptions preferred for generalization |
| Grasp/contact labels | Contact-rich manipulation | Binary contact + force magnitude; timestamped to ≤10ms |
| Success/failure labels | Filtering and reward learning | Per-trajectory binary + failure-mode taxonomy |
Skipping any of these forces your team to build annotation tooling or accept noisier supervision. Both cost more than paying for the annotation upfront.
Collection frequency (Hz) and trajectory length
He et al. (arXiv:2510.21391v1) note that action chunking architectures (ACT, diffusion policy) are sensitive to the temporal resolution of demonstrations. Their results suggest 15–30 Hz action recording balances information density against storage cost for manipulation tasks lasting 5–30 seconds. Below 10 Hz, chunked policies lose the micro-corrections that distinguish success from failure on tight-tolerance tasks. Above 50 Hz, marginal information per frame drops while storage costs scale linearly.
Edge-case distribution
Edge-case distribution is the single most differentiating spec when evaluating a physical AI training data provider. A dataset with 10,000 nominal trajectories and 200 edge cases trains a worse policy than one with 5,000 nominal trajectories and 2,000 edge cases, assuming the edge cases cover your deployment distribution. Require your provider to specify the edge-case ratio and edge-case taxonomy (lighting extremes, occlusion levels, object deformation states, adversarial placements). If they can't enumerate their edge cases, they didn't design for them.
When sim closes the gap alone (and when it can't)
Simulation closes the sim-to-real gap to near-zero for rigid-body locomotion and quasi-static pick-and-place tasks, but fails for contact-rich manipulation involving deformable objects or tight-tolerance insertions. NVIDIA's Isaac Lab and DeepMind's MuJoCo-based pipelines remain the backbone of locomotion policy training, where rigid-body dynamics are well-modeled and visual fidelity matters less. Boston Dynamics and Agility Robotics both use heavy sim pre-training for bipedal locomotion, as described in their ICRA presentations.
The sim-to-real gap narrows to near-zero in specific conditions:
- The task is quasi-static or rigid-body dominant. Pick-and-place of rigid objects on flat surfaces transfers well.
- Visual diversity is the primary gap. Domain randomization plus neural rendering close this.
- The action space is low-dimensional. Navigation, whole-body locomotion, and coarse grasping transfer reliably from sim.
The gap stays wide when:
- Contact dynamics are complex. Deformable objects, granular materials, and multi-finger manipulation all involve physics that MuJoCo's soft contact model and Isaac Sim's FEM solver model with systematic bias, requiring real-world fine-tuning to correct.
- Sensor noise patterns matter. Real force/torque sensors have hysteresis, drift, and cross-axis coupling that sim models poorly.
- The task requires long-horizon reasoning over physical state. Cooking, cable routing, and folding all suffer from compounding sim errors that make demonstrations unreliable past approximately 20 steps.
If you're working on VLA architectures targeting household manipulation, real-world data isn't optional — it's the primary training signal, with sim used for pre-training and augmentation. The question isn't whether to use real data, but how to source it without exceeding your budget.
Evaluating a physical AI training data provider
A physical AI training data provider is a company that collects, annotates, and delivers real-world demonstration datasets for training robot learning policies across calibrated multi-modal sensor streams. Score each criterion below 0–2 (0 = absent, 1 = partial, 2 = meets spec). A provider scoring below 10 out of 16 will likely create more integration work than they save.
| Criterion | What to ask | Weight |
|---|---|---|
| Multi-modal sensor streams (RGB-D, F/T, proprio) | "Which sensors, what calibration, and what sync protocol?" | 2 |
| Collection Hz ≥15 for actions, ≥30 for video | "What is your recording frequency for each stream?" | 2 |
| Temporal synchronization <5ms across modalities | "What is your sync error budget?" | 2 |
| Annotation schema matches your pipeline | "Do you provide 6-DoF pose, segmentation, language labels?" | 2 |
| Edge-case ratio ≥15% with enumerated taxonomy | "What fraction of trajectories are edge cases and how do you define them?" | 2 |
| Embodiment coverage or custom embodiment support | "Which robots have you collected on? Can you match my morphology?" | 2 |
| Data format compatibility (RLDS, HDF5, custom) | "What format do you deliver in?" | 1 |
| Scalable collection (>1,000 trajectories/week) | "What is your throughput and how many operators run in parallel?" | 1 |
Claru operates in this space as a physical AI training data provider. Their teams of trained human operators collect embodied AI datasets across multiple robot platforms, delivering synchronized RGB-D + force/torque + proprioception streams at 30 Hz with <3ms cross-modal sync error, along with 6-DoF pose annotations and language task labels. For teams building VLA training pipelines that need contact-rich demonstration data beyond what sim provides, this kind of turnkey collection eliminates 4–8 weeks of in-house infrastructure buildout.
Toyota Research Institute (TRI) is one of the few organizations that has publicly described operating an in-house large-scale real-world data collection pipeline for manipulation. Most frontier labs, including those at Google DeepMind and Physical Intelligence, supplement internal collection with external data partnerships, though specific provider relationships are rarely disclosed.
The honest answer: if your task is rigid-body pick-and-place in a constrained environment, you probably don't need an external data provider. Save your budget. But if you're building a general-purpose manipulation policy that must handle deformables, tool use, or multi-step assembly, real-world data at the right spec is the highest-ROI investment you can make — more impactful per dollar than an extra GPU node.
Takeaways
- The RoboBrain survey by Ji et al. (arXiv:2412.09632v2) identifies sensor modality diversity, not trajectory count, as the primary bottleneck for robot foundation models.
- He et al. (arXiv:2510.21391v1) show that heterogeneous real demonstrations from 5 embodiment types match or beat 20× more sim trajectories from a single embodiment on held-out manipulation tasks.
- Real-world force/torque data at ≥100 Hz is required for contact-rich tasks; sim contact models introduce >40% force residual errors on tight-tolerance insertions, per He et al. (arXiv:2510.21391v1).
- Sim closes the gap for rigid-body tasks, locomotion, and visual diversity augmentation. It fails on deformable manipulation, granular materials, and long-horizon physical reasoning.
- A physical AI training data provider should deliver ≥15 Hz multi-modal streams with <5ms cross-sensor synchronization and an edge-case ratio above 15%.
- Action chunking architectures (ACT, diffusion policy) degrade below 10 Hz demonstration frequency, per He et al. (arXiv:2510.21391v1).
- Before committing to a real-data buy, run 50+ real-world eval rollouts and classify failures. If fewer than 15% require real data, invest in sim improvements instead.
FAQ
What is a physical AI training data provider?
A physical AI training data provider is a company that collects, annotates, and delivers real-world demonstration datasets for training robot learning policies — typically VLAs, diffusion policies, or other architectures that control physical systems. Unlike synthetic data companies that generate trajectories in simulation, these providers operate physical robots or human teleoperation rigs in real environments, capturing synchronized multi-modal sensor streams: RGB-D video, force/torque readings, proprioceptive joint states, and sometimes tactile arrays. The distinction from generic data labeling companies is expertise in temporal synchronization, sensor calibration, and the annotation schemas (6-DoF pose, contact labels, language task descriptions) that robot learning pipelines require. Claru, Scale AI (via its robotics vertical), and several smaller startups operate in this space, alongside in-house teams at labs like Toyota Research Institute.
How much real-world robot data do I need to close the sim-to-real gap?
He et al. (arXiv:2510.21391v1) show that as few as 200–500 real-world trajectories can measurably improve contact-rich task performance when used to fine-tune a sim-pretrained policy. However, diminishing returns set in at different points for different tasks. For rigid pick-and-place, 100–300 real demonstrations often suffice. For deformable manipulation or multi-step assembly, teams at leading labs have reported needing 2,000–10,000 demonstrations before plateauing. The data volume question is best answered empirically: collect a small pilot batch (200–500 trajectories), measure your eval improvement slope, and extrapolate. A steep slope means scale the collection. A flat slope means your bottleneck is elsewhere — likely architecture or reward specification.
What sensor modalities should robot training data include?
The RoboBrain survey by Ji et al. (arXiv:2412.09632v2) identifies RGB-D (color plus calibrated depth), force/torque at the end-effector, and full proprioceptive joint state as the minimum viable sensor suite for general manipulation policies. Depth must be factory-calibrated to the RGB frame; post-hoc alignment introduces errors that compound during policy rollout. Force/torque should be sampled at ≥100 Hz to capture contact transients, even if your policy only consumes 15–30 Hz action labels (you can downsample but you can't upsample). Adding wrist-mounted and third-person cameras together improves bimanual task performance by 10–18% over either viewpoint alone in ALOHA-style setups, according to results reported in the Open X-Embodiment project. Tactile sensing (GelSight, DIGIT) is increasingly useful for in-hand manipulation but isn't standard in most embodied AI datasets yet. If your task involves discriminating surface properties or detecting incipient slip, require tactile streams from your provider.
Is synthetic data enough for training physical AI models?
Synthetic data is sufficient for rigid-body locomotion and visual perception pre-training, but insufficient for contact-rich manipulation involving deformable objects or tight-tolerance insertions. Locomotion policies from Boston Dynamics and Agility Robotics transfer from sim to real with minimal or zero real-world fine-tuning, because rigid-body dynamics are modeled accurately and the action space is low-dimensional. Visual perception modules for object detection and pose estimation can be pre-trained almost entirely on synthetic renders from NVIDIA Isaac Lab with aggressive domain randomization. But for contact-rich manipulation — especially involving deformable objects, granular materials, or tight-tolerance insertions — sim-only training produces policies that fail at rates 30–60% higher than sim+real pipelines, per results aggregated by He et al. (arXiv:2510.21391v1). The sim-to-real gap persists wherever the physics engine's contact model diverges from reality, and no amount of domain randomization fixes a systematically biased simulator.
How do I evaluate the quality of a robot training dataset?
Five checks determine whether a robot training dataset meets production quality standards. First, verify temporal sync error between sensor modalities — anything above 5ms introduces artifacts in action-chunking policies. Second, check annotation accuracy by requesting sample 6-DoF pose labels and verifying against your own measurements; translation error should be below 5mm and rotation error below 3°. Third, confirm the edge-case ratio — at least 15% of trajectories should cover non-nominal conditions (lighting extremes, occlusion, unusual object placements), with a documented taxonomy. Fourth, require success/failure labels with failure-mode classification, so you can filter or weight trajectories during training. Fifth, verify format compatibility — confirm the dataset ships in RLDS, HDF5, or whatever your data enrichment pipeline consumes without custom conversion scripts. If a provider can't answer these questions with specific numbers, they're selling video, not training data.