Zero-Shot Manipulation: Handling Novel Objects Without Task-Specific Training

Zero-shot manipulation is a robot's ability to perform manipulation tasks involving objects, configurations, or environments that were not present in its training data. The term 'zero-shot' indicates that the policy receives zero training examples of the specific test scenario — it must generalize entirely from experience with other objects and tasks.

This capability is critical for real-world deployment because no training dataset can cover every object a robot will encounter. A warehouse robot will face novel product packaging, a kitchen robot will encounter unfamiliar utensils, and a home assistant robot will need to handle personal items unique to each household. Zero-shot manipulation is the bridge between laboratory demonstrations and practical utility.

The technical foundation rests on learning representations that capture general manipulation-relevant features rather than object-specific memorization. Shape, surface normals, material properties, affordances (where to grasp, where to push), and physical dynamics must be inferred from visual and tactile observation without prior knowledge of the specific object. Modern approaches achieve this through massive training data diversity, pretrained vision encoders, and language-conditioned policies that enable compositional generalization.

For teams building zero-shot manipulation systems, the primary practical lever is training data diversity. The marginal value of an additional demonstration of an existing object diminishes rapidly, while demonstrations of novel objects have high marginal value. Data collection protocols should prioritize breadth: many objects, many environments, many initial configurations. Claru's data collection operations target exactly this diversity, providing demonstrations across hundreds of object categories with controlled variation in pose, lighting, and background.

Evaluation protocol design is crucial. The held-out test set must be genuinely novel — objects that differ meaningfully from training objects, not just new instances of the same category. Best practice is to define three evaluation tiers: in-distribution (same objects as training), category-level zero-shot (new instances of training categories), and cross-category zero-shot (entirely new categories). Reporting success rates at each tier gives a realistic picture of generalization capability.

For production deployment, zero-shot manipulation systems should be paired with failure detection and recovery mechanisms. Even the best zero-shot systems fail on some objects, and the failure rate increases for objects further from the training distribution. Real-time grasp confidence estimation, force-based failure detection, and retry strategies with varied grasp poses make zero-shot systems practical for deployment rather than purely academic evaluation.

Claru's data collection operations prioritize object diversity — the single most important factor for zero-shot manipulation performance. Our datasets cover hundreds of object categories across household, industrial, and commercial domains, with controlled variation in pose, lighting, and scene configuration. This diversity is what enables policies trained on Claru data to generalize to objects they have never seen.

Zero-shot manipulation refers to a robot policy's ability to perform manipulation tasks on objects, configurations, or environments not present in the training data. In machine learning terminology, 'zero-shot' means the policy receives zero examples of the specific test scenario during training. A robot trained to pick up mugs, bowls, and bottles that can also pick up a stapler it has never encountered is demonstrating zero-shot manipulation.

The challenge is fundamentally about generalization. Traditional robot manipulation systems are engineered for specific objects with known geometry — an automotive assembly line robot is programmed with the exact CAD model of every part it handles. Zero-shot manipulation requires the robot to perceive novel geometry, infer graspable surfaces, predict physical properties like weight and friction, and plan motions that avoid collisions — all without prior experience with the target object.

Two distinct approaches enable zero-shot manipulation. The first is large-scale data diversity: by training on demonstrations spanning thousands of different objects, the policy learns general manipulation primitives that transfer to novel objects. RT-2 (Google DeepMind, 2023) demonstrated this approach, using a VLA model trained on thousands of objects to manipulate novel items specified by natural language instructions. The second is explicit geometric reasoning: systems like AnyGrasp compute grasps directly from point clouds using learned geometric features, without requiring any task demonstrations.

The most reliable path to zero-shot manipulation is training data diversity. A policy trained on 100 demonstrations of picking up one specific mug will fail on novel objects. A policy trained on 10,000 demonstrations across 500 different objects — varying in shape, size, material, color, and weight — learns general features that transfer. The critical insight is that diversity in training objects matters more than volume per object. Ten demonstrations each across 1,000 objects outperforms 10,000 demonstrations of a single object for zero-shot tasks.

Language conditioning dramatically improves zero-shot manipulation by providing a compositional interface for specifying novel tasks. Instead of training a separate policy for 'pick up the red ball' and 'pick up the blue cube', a language-conditioned policy learns to compose color, shape, and action concepts. At test time, it can handle 'pick up the green cylinder' — a combination never seen in training — because it has learned the individual concepts. This requires training data with diverse language instructions paired with diverse objects and actions.

Synthetic data augmentation extends zero-shot capabilities by exposing the policy to procedurally generated object variations. Tools like Blender and Isaac Sim can create thousands of object meshes with varied geometry, texture, and physical properties. Policies trained on a mixture of real demonstrations and synthetic renderings achieve better zero-shot performance than either alone. The key is careful domain randomization: varying textures, lighting, camera positions, and background clutter so the policy cannot overfit to any single visual distribution.

Evaluating zero-shot manipulation requires careful protocol design. The standard approach is a held-out object test: train on a set of objects, test on a disjoint set the policy never saw. Success rate on held-out objects is the primary metric. More rigorous evaluations also measure category-level generalization (training on mugs, testing on mugs with very different shapes) versus cross-category generalization (training on kitchenware, testing on tools).

SIMPLER (Li et al., 2024) and RLBench (James et al., 2020) provide standardized benchmarks for zero-shot manipulation evaluation. SIMPLER includes Google Robot and WidowX environments with curated held-out object sets. RLBench defines 100 manipulation tasks with procedurally varied initial configurations. For real-world evaluation, teams typically define a set of 20-50 novel household objects spanning diverse categories and report grasp and task success rates across 10+ trials per object.

Common failure modes in zero-shot manipulation include transparent objects (depth sensors and vision models struggle with glass), deformable objects (bags, cloth), very small objects (below the gripper's minimum aperture), and extremely heavy objects (exceeding the robot's payload). Understanding these failure modes is essential for scoping the practical deployment envelope of a zero-shot manipulation system and for targeting data collection efforts at the boundary cases.