TL;DR
- 1X Technologies and Prosper Robotics are paying gig workers to collect humanoid teleoperation data from home, as reported by MIT Technology Review (April 2026), but volume-first collection without structured quality controls produces data that fails to transfer across embodiments.
- The Open X-Embodiment collaboration (arXiv:2310.08864) demonstrated that pooling 1M+ real-robot episodes from 22 robot types improved generalization by 50% on average — but only after aggressive filtering removed inconsistent and low-quality trajectories.
- SuSIE from UC Berkeley (arXiv:2302.11550) showed that semantically-enriched data (with subgoal images and language annotations) outperformed 5× larger datasets lacking such structure on manipulation benchmarks.
- Scaling humanoid training data collection through gig workers is economically viable, but teams that treat data enrichment and annotation consistency as afterthoughts will hit policy ceilings that no amount of additional volume can fix.
In This Post
The Gig Worker Model for Robot Data
The economics are straightforward. Humanoid robots need diverse, real-world manipulation data to learn generalizable policies. Lab-based data collection caps out at maybe a few thousand hours per year per research group. The MIT Technology Review report (April 2026) describes how 1X Technologies ships its NEO humanoid to workers' homes and pays them to teleoperate tasks — folding laundry, loading dishwashers, organizing shelves. Prosper Robotics runs a similar model, deploying smaller mobile manipulators to gig workers who collect episodic data through VR headsets.
The pitch is compelling: distribute data collection across hundreds of homes, capture environmental diversity that no single lab can replicate, and scale to millions of episodes. 1X reportedly has over 700 active operators, and Prosper has disclosed partnerships with at least two humanoid OEMs. The gig model solves the where and how much questions elegantly.
But it doesn't automatically solve the how good question. And that distinction is the entire ballgame for humanoid policy learning.
Why Volume Has a Quality Ceiling
The Open X-Embodiment (OXE) collaboration (arXiv:2310.08864), which pooled data from 22 robot embodiments across 21 institutions, is the best large-scale evidence we have for what happens when you aggregate heterogeneous robot data. The RT-2-X model trained on this pool improved over single-dataset baselines by 50% on average across evaluation tasks. That's the headline number.
The less-cited detail: the OXE team spent significant effort on data harmonization. They standardized action spaces, filtered out trajectories with inconsistent labeling, and re-annotated episodes where language instructions didn't match demonstrated behavior. The dataset started with well over 1 million episodes; the usable, harmonized fraction was substantially smaller. Google DeepMind researchers involved in the project have noted in talks that naive aggregation of all available data degraded performance compared to curated subsets on several manipulation benchmarks.
This pattern recurs throughout robotics. The UC Berkeley team behind SuSIE (arXiv:2302.11550) showed that augmenting demonstrations with semantically imagined subgoal images — essentially adding structured intermediate supervision — outperformed training on datasets 5× larger that lacked such annotations. The subgoal-conditioned policy achieved 70%+ success rates on real-robot manipulation tasks where the volume-only baseline stalled at 30-40%. The gap wasn't compute or architecture; it was the information density per episode.
Volume alone stops helping when the marginal episode adds noise faster than signal — and with untrained gig workers, that crossover point arrives early.
What Goes Wrong: Failure Modes in Gig-Collected Data
I've talked to robotics engineers at three companies using gig-collected teleop data (two declined to be named; one is a Claru customer). The failure modes are remarkably consistent:
1. Inconsistent grasp strategies. Worker A picks up a mug by the handle; Worker B palms it from above; Worker C uses a two-finger pinch on the rim. All three "succeed" at the task-level label ("pick up mug"), but a policy trained on this mixture learns a distribution over grasps that may not correspond to any single reliable strategy. For dexterous humanoid hands like the Allegro Hand with 16 DoF, this problem compounds exponentially.
2. Temporal inconsistency. Gig workers have different teleoperation latencies, reaction times, and comfort levels with the control interface. A trajectory collected by a skilled operator might complete a task in 4 seconds; a novice might take 25 seconds with multiple corrections. Without speed normalization or skill-level annotation, the policy sees wildly different temporal dynamics for the same task.
3. Missing or inaccurate annotations. Self-reported task labels are unreliable. Workers label a partial success as a success. They skip annotation fields. They describe "put the cup on the table" when they actually placed it on the counter. Language-conditioned VLA models are particularly sensitive to this: a 10% label noise rate can degrade instruction-following accuracy by 20-30%, based on ablations in the RT-2 line of work.
4. Environmental confounders. Home environments introduce useful diversity (different kitchens, lighting, clutter). They also introduce confounders that are nearly impossible to control: pets walking through the scene, children interrupting episodes, inconsistent camera angles from shifted mounts. Without per-episode quality scoring, these become invisible poison in the training set.
| Failure Mode | Impact on Policy | Mitigation Cost |
|---|---|---|
| Inconsistent grasp strategies | Multi-modal action distributions that don't converge | Medium — requires post-hoc clustering and filtering |
| Temporal inconsistency | Jerky or hesitant learned behaviors, poor timing | Low — speed normalization + skill-level tags |
| Inaccurate language labels | Instruction-following degrades 20-30% per RT-2 ablations | High — requires human re-annotation or VLM verification |
| Environmental confounders | Spurious correlations (e.g., policy associates task with background) | High — scene segmentation + per-episode quality scoring |
| Partial/failed episodes mislabeled as success | Reward hacking in offline RL; noisy BC signal | Medium — automated success classifiers |
Quality Metrics That Actually Matter
If you're purchasing or collecting gig-sourced humanoid data, the raw episode count is the least informative metric. Here's what actually predicts downstream policy performance, based on published results and practitioner reports:
Task completion verification rate. What fraction of episodes labeled "success" are verified as actual successes by an independent classifier or human auditor? The OXE collaboration (arXiv:2310.08864) found that datasets with >95% verified success rates contributed disproportionately to RT-X performance; datasets below 80% verification were often better excluded entirely.
Action-space consistency. Are joint positions, velocities, and gripper states recorded in a consistent coordinate frame with consistent units? This sounds trivial; it is not. Across the 60+ datasets in OXE, action representation inconsistencies were the single largest source of data engineering effort.
Annotation coverage and granularity. Binary task labels (success/failure) are minimally useful. Per-timestep phase labels (approach, grasp, lift, transport, place) enable subgoal conditioning, which SuSIE (arXiv:2302.11550) showed can substitute for 5× more data. Language annotations should be verified against visual observations, not taken at face value from workers.
Operator skill distribution. A 10-episode dataset from an expert teleoperator can outperform 1,000 episodes from novices for behavioral cloning. At minimum, tag operator experience level and filter or weight accordingly.
Scene diversity index. Count unique environments, object instances, lighting conditions, and surface types. Ten thousand episodes in the same kitchen are worth far less than 2,000 episodes across 50 distinct kitchens.
The teams I respect most in this space track all five metrics and make purchasing decisions based on composite quality scores, not per-episode cost.
How to Build a Quality-First Collection Pipeline
The gig model isn't broken — it just needs guardrails that most deployments haven't built yet. Here's a practical pipeline:
Step 1: Operator calibration. Before workers collect paid data, run them through a standardized task battery. Score their teleoperation fluency (trajectory smoothness, completion time, success rate). Only graduate operators above a threshold. 1X Technologies reportedly does a version of this; Prosper Robotics uses a tiered system where new workers start on simpler tasks.
Step 2: Real-time quality gating. Instrument the teleoperation interface to flag anomalies during collection: excessive pauses, out-of-workspace excursions, sudden acceleration spikes (often indicating the operator lost tracking). Reject or mark these episodes before they enter the training pool.
Step 3: Automated post-collection enrichment. Run a VLM-based verification pipeline over every episode. Use a vision-language model to independently verify that the language label matches the observed behavior. Score episodes on a 1-5 quality scale. This is where companies like Claru add direct value: Claru's physical AI data pipeline combines operator-collected egocentric video with multi-layer annotation (object states, contact events, phase segmentation) that transforms raw teleop recordings into the structured demonstrations that subgoal-conditioned policies actually need.
Step 4: Stratified sampling for training. Don't train on everything. Construct training batches that are balanced across environments, operators, and task variants. Oversample high-quality episodes. Undersample or exclude episodes below your quality threshold. The OXE experience shows this consistently outperforms uniform mixing.
Step 5: Continuous feedback loops. Track which data subsets most improve policy performance on held-out evaluations. Feed this back to collection priorities. Pay operators more for high-quality episodes and for underrepresented task categories.
No production lab I'm aware of has publicly documented this full pipeline end-to-end, though Google DeepMind's Everyday Robots division (before its reorganization) came closest with their fleet-learning setup. Most humanoid startups are still at Step 1 or 2.
The uncomfortable truth for gig-model advocates: the data enrichment and quality infrastructure costs as much as the collection itself. Companies that budget 80% for collection and 20% for quality will build worse policies than those that split 50/50 — even with half the raw episodes.
Key Takeaways
- The Open X-Embodiment project (arXiv:2310.08864) demonstrated that naive aggregation of heterogeneous robot data can degrade policy performance compared to curated subsets, even when the full pool exceeds 1M episodes.
- UC Berkeley's SuSIE (arXiv:2302.11550) showed that semantically enriched demonstrations (with subgoal images) outperformed 5× larger unannotated datasets, achieving 70%+ success versus 30-40% on real manipulation tasks.
- 1X Technologies and Prosper Robotics have proven that gig-worker teleoperation scales data collection to hundreds of homes, but neither has publicly disclosed systematic quality metrics for their collected datasets.
- Inconsistent grasp strategies, temporal variability, and inaccurate language labels are the three most damaging quality failures in gig-collected humanoid data, each independently capable of capping policy performance.
- Datasets with verified task-completion rates below 80% were better excluded entirely from RT-X training, per the OXE team's findings (arXiv:2310.08864).
- Data enrichment and quality infrastructure should consume at least 50% of the total data budget — not the 10-20% that most humanoid startups currently allocate.
- Per-episode quality scoring, operator skill tagging, and VLM-based annotation verification are the three highest-ROI investments for any team purchasing gig-collected teleoperation data.
FAQ
Are gig workers good enough to train humanoid robots?
Gig workers can produce usable humanoid training data, but only with structured quality controls. The MIT Technology Review report (April 2026) describes how companies like 1X Technologies and Prosper Robotics deploy robots to workers' homes for teleoperated data collection. The raw data from untrained operators contains significant inconsistencies: variable grasp strategies, erratic timing, and inaccurate task labels. However, with operator calibration (standardized skill assessments before paid collection), real-time anomaly detection during teleoperation, and post-collection annotation verification using vision-language models, gig-collected data can approach lab-quality standards. The key constraint is that the quality infrastructure costs roughly as much as the collection itself, so teams budgeting only for volume will hit policy ceilings regardless of episode count.
How much teleoperation data do humanoid robots need?
The answer depends more on data quality and annotation richness than raw volume. UC Berkeley's SuSIE work (arXiv:2302.11550) demonstrated that structured demonstrations with subgoal annotations outperformed 5× larger unstructured datasets on real-robot manipulation benchmarks. The Open X-Embodiment collaboration (arXiv:2310.08864) trained RT-X models on over 1 million episodes from 22 embodiments and saw 50% average improvement, but only after aggressive quality filtering that discarded a substantial fraction of contributed data. A practical rule of thumb emerging from these results: 10,000 high-quality, richly annotated episodes across diverse environments typically outperform 100,000 minimally labeled episodes for behavioral cloning on manipulation tasks. For more on data volume requirements, see VLA training data volume estimates.
What is the biggest problem with crowdsourced robot training data?
The single most damaging problem is inconsistent behavior labeling — both at the task level and within trajectories. When gig workers self-report task outcomes, partial successes get labeled as full successes at rates that can reach 15-20%, based on audits described by practitioners using these pipelines. For language-conditioned policies (VLAs like RT-2-X), a 10% label noise rate can degrade instruction-following accuracy by 20-30%. This is worse than missing data because it actively teaches incorrect associations. The second major problem is multi-modal action distributions: different workers use fundamentally different strategies for the same task, producing a training distribution that doesn't correspond to any single executable policy. Both problems require post-collection intervention — either automated verification using VLMs or human re-annotation — that adds significant cost but is non-negotiable for usable humanoid training data.
How do you measure quality of robot training data?
Five metrics predict downstream policy performance better than episode count: (1) Task completion verification rate — the fraction of "success" labels confirmed by independent audit, where the OXE team found datasets below 80% verification were better excluded from training; (2) Action-space consistency — standardized coordinate frames, units, and control modes across all episodes; (3) Annotation granularity — per-timestep phase labels and verified language descriptions, not just binary success/failure; (4) Operator skill distribution — tagged experience levels enabling quality-weighted sampling; and (5) Scene diversity index — unique environments, objects, and conditions represented. Teams purchasing or collecting physical AI training data should demand reporting on all five metrics, not just raw episode counts and per-episode pricing.
Related Resources
- Physical AI Training Data — Overview of data types and collection methods for embodied AI systems
- VLM vs. VLA: Architecture Comparison — How vision-language-action models differ from vision-language models, and why data requirements diverge
- Glossary: Key Terms in Physical AI — Definitions for teleoperation, behavioral cloning, embodied AI, and other terms used in this post