Object Rearrangement Training Data

Claru builds rearrangement datasets using modular scene fixtures (shelving units, counters, desk surfaces) shipped to our distributed collection network. Each episode captures paired initial and goal states with multi-view RGB-D, natural language goal descriptions, and full manipulation trajectories via teleoperation. Our object library spans 200+ items across grocery, kitchenware, office, and household categories with calibrated meshes for pose estimation ground truth. Annotations include per-object placement accuracy, ordering efficiency scores, collision events, and scene complexity metadata. We deliver difficulty-stratified datasets in RLDS, HDF5, or custom formats with splits by object count, scene type, and goal specification modality. Typical campaigns produce 5,000-20,000 annotated episodes within 3-6 weeks.

Object rearrangement is the task of moving a set of objects from an initial configuration to a target arrangement. It sounds simple, but it sits at the intersection of perception (identifying what needs to move), planning (determining the order and placement), and manipulation (executing picks and places without collisions). A robot tidying a kitchen counter must recognize which items are out of place, reason about where each belongs (plates in the rack, cups on the shelf, utensils in the drawer), and execute a sequence of grasps and placements that avoids toppling or breaking anything.

The combinatorial complexity makes rearrangement a data-hungry task. With N objects, there are N! possible placement orderings, and each placement requires spatial reasoning about collision-free approach trajectories. The Habitat Rearrangement Challenge (Szot et al., 2021) showed that even with millions of simulation episodes, agents struggle with multi-object rearrangement in unseen environments. Real-world data is essential because simulation cannot faithfully capture cluttered scenes with diverse object geometries, surface friction, and visual appearance at production quality.

Commercial interest in rearrangement spans retail inventory (restocking shelves at $11B annual labor cost), warehouse fulfillment (object putaway after receiving), home robotics (tidying rooms), and manufacturing (kitting parts for assembly). Each application demands a different notion of the goal state — retail requires planogram compliance, warehousing optimizes for access frequency, and home tidying follows implicit cultural norms. Training data must encode these domain-specific goal specifications alongside the manipulation demonstrations.

The personalization dimension is uniquely important for rearrangement. Every household organizes differently — one person's 'tidy' kitchen is another person's disorganized mess. TidyBot (Wu et al., 2023) showed that LLMs can infer organizational preferences from a handful of examples, but the underlying manipulation policy still needs diverse training data to handle the physical act of rearranging objects across varied surfaces and containers. This creates a two-tier data requirement: a large general manipulation dataset (5K-50K episodes) that teaches the robot how to pick and place diverse objects, plus a per-user personalization dataset (50-200 episodes) that teaches where each user wants things placed. Both tiers are necessary for deployment-grade rearrangement.

The field has progressed from single-object pick-and-place to multi-object sequential rearrangement. SayCan (Ahn et al., 2022) chains language model planning with learned manipulation primitives to execute rearrangement instructions like 'put all the fruit in the bowl,' scoring 84% success on seen objects but dropping to 47% on novel arrangements. StructFormer (Liu et al., 2022) predicts the spatial structure of target arrangements using transformer architectures, achieving 78% accuracy on multi-object placements by learning structural priors from 10,000+ training scenes.

DALL-E-Bot (Kapelyukh et al., 2023) introduces a novel approach: using text-to-image models to generate goal images from language descriptions, then using visual difference between current and goal states to plan rearrangement actions. This decouples goal specification from manipulation, but requires high-quality paired data (current state, goal state, action sequence) for training the manipulation policy. The data requirement shifts from goal diversity (handled by the generative model) to manipulation diversity (varied grasps, placements, and obstacle avoidance).

The current bottleneck is long-horizon rearrangement — scenes with 10+ objects requiring 20+ sequential manipulation steps. Error compounds across steps: a 95% per-step success rate yields only 36% success over 20 steps. TidyBot (Wu et al., 2023) addresses this by learning personalized tidying preferences from user demonstrations, reducing the planning search space. The data implication is that rearrangement datasets need not just diverse scenes but diverse users whose organizational preferences create natural variety in goal states.

For retail applications, rearrangement data must capture planogram compliance — placing products in exact positions defined by a store layout document. This requires sub-centimeter placement accuracy and front-facing orientation for labeled products. The data annotation must include not just where objects are placed but their orientation (label facing forward), spacing (equal gaps between products), and depth (pushed to the back of the shelf). Retail rearrangement data is uniquely structured because the goal state is precisely specified by the planogram rather than approximately specified by an image or language description, enabling quantitative placement accuracy metrics that other rearrangement domains lack.

Rearrangement data collection uses a paired-scene protocol. For each episode: (1) arrange the scene in its goal state and capture multi-view images plus object poses, (2) disturb the scene by randomly displacing 3-10 objects, (3) record the demonstration as the operator rearranges objects back to the goal state via teleoperation. This produces matched (initial state, goal state, trajectory) triplets. The goal state images serve as conditioning for goal-conditioned policies, while the language descriptions of the arrangement serve language-conditioned approaches.

Object diversity is the primary scaling axis. A production rearrangement dataset should cover 50-200 unique object types spanning categories relevant to deployment: groceries, kitchenware, office supplies, household items. Each object needs consistent mesh or point cloud models for pose estimation ground truth. Scene complexity should be stratified: 30% of episodes with 3-5 objects (easy), 40% with 6-12 objects (medium), and 30% with 13-25 objects (hard) to enable curriculum learning.

Claru's rearrangement collection protocol uses modular scene templates — standardized shelving units, desk surfaces, and counter spaces that are physically shipped to collection sites. Operators receive target arrangement photos and natural language descriptions before each episode. Post-hoc annotations include per-object placement accuracy (centimeters from target pose), ordering efficiency (how close to optimal was the sequence), and collision events during manipulation. All episodes include synchronized RGB-D from 3+ views, robot proprioception, and language descriptions.