Training Data for Covariant
Covariant is building advanced robotic systems. Here is how real-world data accelerates their development from prototype to production deployment.
About Covariant
Covariant builds AI-powered robotic systems for warehouse automation and logistics. Founded in 2017 by Pieter Abbeel, Peter Chen, and Rocky Duan from UC Berkeley, the company has raised over $222 million and deploys its Covariant Brain AI platform across warehouse and logistics operations globally. Their system enables robots to pick, place, and sort diverse objects in unstructured environments.
Covariant at a Glance
Known Data Requirements
Covariant's foundation model approach RFM-1 demands massive volumes of real-world manipulation data across diverse object types, bin configurations, and warehouse environments. Their deployment across logistics facilities generates proprietary data, but scaling to general-purpose manipulation requires far broader object and environment diversity than any single warehouse network provides.
Diverse object grasping demonstrations
Source: RFM-1 foundation model announcement, 2024
Grasping demonstrations across thousands of object categories with varying shapes, materials, weights, and packaging types. RFM-1 needs broad coverage of the long-tail distribution of warehouse SKUs.
Multi-camera bin picking recordings
Source: Covariant warehouse deployment case studies
Multi-view recordings of bin picking operations showing objects in cluttered arrangements, partial occlusion, and varied lighting conditions that real warehouse environments present.
Egocentric warehouse activity video
Source: Covariant hiring for perception research, 2024
First-person video of human warehouse workers performing pick, pack, and sort operations to pretrain visual features for Covariant Brain's perception system.
How Claru Data Addresses These Needs
| Lab Need | Claru Offering | Rationale |
|---|---|---|
| Diverse object grasping demonstrations | Manipulation Trajectory Dataset + Custom Collection | Claru's manipulation data covers diverse object interactions with multi-modal annotations. Custom campaigns can target specific SKU categories and packaging types Covariant needs for RFM-1 training. |
| Multi-camera bin picking recordings | Multi-View Manipulation Dataset | Synchronized multi-camera recordings of manipulation tasks provide the viewpoint diversity that Covariant's perception system needs for robust bin picking across camera configurations. |
| Egocentric warehouse activity video | Egocentric Warehouse Video Dataset | Purpose-collected first-person video from real warehouse operations across 100+ facilities provides the visual pretraining data that grounds Covariant Brain in real logistics environments. |
Technical Data Analysis
Covariant's RFM-1 represents one of the most ambitious attempts to build a foundation model specifically for robotic manipulation. Unlike VLA models that adapt vision-language models for robotics, RFM-1 is designed from the ground up for physical interaction — learning the physics of grasping, the geometry of bin picking, and the logistics of warehouse operations.
The foundation model approach creates an insatiable appetite for data diversity. While Covariant's deployed robot fleet generates continuous proprietary data, this data comes from specific warehouse configurations with specific object distributions. The long-tail problem is acute: warehouses handle millions of unique SKUs, and a foundation model must handle objects it has never seen before. This requires training data that covers the full diversity of object shapes, materials, weights, and packaging types — far beyond what any single warehouse network encounters.
Covariant's sim-to-real pipeline for grasping handles rigid objects well but struggles with deformable items (bags, pouches, clothing) and transparent packaging (blister packs, shrink-wrapped items). Real-world grasping data for these challenging object categories provides the fine-tuning signal that closes the sim-to-real gap for the hardest cases.
The warehouse environment itself varies significantly across deployments. Lighting conditions, bin types, conveyor configurations, and ambient temperature all affect perception and grasp success. Training data from diverse real warehouse environments ensures RFM-1 develops robust features that transfer across deployment sites rather than overfitting to the visual characteristics of any single facility.
Key Research & References
- [1]Covariant Team. “RFM-1: A Robotics Foundation Model.” Covariant Blog, 2024. Link
- [2]Kalashnikov et al.. “Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation.” CoRL 2018, 2018. Link
- [3]Open X-Embodiment Collaboration. “Open X-Embodiment: Robotic Learning Datasets and RT-X Models.” ICRA 2024, 2024. Link
Frequently Asked Questions
Covariant's foundation model approach RFM-1 demands massive volumes of real-world manipulation data across diverse object types, bin configurations, and warehouse environments. Their deployment across logistics facilities generates proprietary data, but scaling to general-purpose manipulation requires far broader object and environment diversity than any single warehouse network provides.
Simulation cannot faithfully model the contact dynamics, material properties, and environmental conditions that Covariant's robots encounter in deployment. Real-world data provides the distributional coverage that fills simulation gaps — authentic surfaces, lighting conditions, and object interactions from actual deployment environments.
Yes. Claru operates a global network of 10,000+ data collectors across 100+ cities who can capture teleoperated demonstrations, egocentric video, and sensor data in target environments using standardized recording protocols.
Accelerate Covariant's Data Pipeline
Talk to our team about purpose-built datasets for Covariant's robotic systems.