Table Setting Training Data

Claru collects table setting data using instrumented tabletop workstations with overhead 4K cameras, precision-calibrated coordinate systems, and inventories of 20+ real tableware sets spanning Western formal, Western casual, Asian, and custom conventions. Each demonstration is annotated with sub-centimeter placement coordinates, relational spatial predicates between items, cultural compliance labels, and placement quality scores. Our operators demonstrate complete place settings following specified cultural rules with natural variation in approach trajectories, and every dataset includes train/validation/test splits stratified by tableware set and cultural style. We deliver in RLDS, HDF5, or Zarr formats compatible with Diffusion Policy, ACT, SayCan, and foundation model architectures, with standard throughput of 100-200 annotated settings per day per workstation.

Table setting is a canonical benchmark for long-horizon spatial reasoning in service robotics. Unlike simple pick-and-place tasks where objects have a single target location, table setting requires understanding relational constraints: the fork goes to the left of the plate, the knife to the right with the blade facing inward, the water glass above the knife. These spatial relationships are culturally determined, context-dependent (formal vs. casual), and must be maintained across varying table shapes, sizes, and existing place settings. A robot that can reliably set a table demonstrates the spatial reasoning capabilities needed for dozens of downstream service tasks.

The challenge compounds because table setting is inherently multi-step and multi-object. A typical formal place setting involves 8-12 items that must be placed in a specific sequence to avoid collisions and occlusion — you cannot place the bread plate after the main plate if the bread plate position overlaps the approach trajectory. SayCan (Ahn et al., 2022) demonstrated that combining LLM planning with learned manipulation primitives enables table-setting-like sequential tasks, but the manipulation policies themselves require demonstration data that captures the precise spatial relationships and gentle placement dynamics for fragile tableware.

Current public datasets for table setting are extremely limited. The BEHAVIOR-1K benchmark (Li et al., 2023) includes table setting as one of 1,000 household activities but provides only simulated demonstrations. The RT-2 evaluation suite tests table setting with a few dozen configurations. Real-world table setting data that captures the diversity of tableware (ceramic plates, glass stemware, metal utensils of varying sizes), table surfaces (wood, glass, cloth-covered), and cultural conventions (Western formal, Japanese kaiseki, Chinese banquet) simply does not exist at scale in any public dataset.

Building production-grade table setting policies requires 1,000-5,000 demonstrations across at least 20 distinct tableware sets and 5+ table configurations. Each demonstration must include precise placement coordinates (sub-centimeter accuracy), the spatial relationships between items (relative positions encoded as relational predicates), and arrangement rule compliance labels (correct/incorrect placement per cultural standard). Without this structured annotation, policies learn to place objects at approximate positions without understanding the relational rules that define a correctly set table.

SayCan (Ahn et al., 2022) demonstrated that grounding LLM planning in learned manipulation affordances enables multi-step tasks like table setting. The system decomposes natural language instructions into sequences of manipulation primitives, with each primitive scored by a learned value function trained on real robot data. For table setting, SayCan successfully planned and executed 8-item place settings by chaining pick-and-place primitives, achieving 74% end-to-end success rate on the full setting sequence. The primary failure mode was accumulating placement error across sequential items rather than individual primitive failure.

TidyBot (Wu et al., 2023) addressed the personalization challenge in household arrangement tasks. By combining a small set of user-specific examples (5-10 demonstrations) with LLM-based category reasoning, TidyBot achieved 85% placement accuracy on novel items following the user's personal organization preferences. For table setting, this approach enables robots to learn household-specific conventions (which side the napkin goes on, whether wine glasses are used) from minimal examples rather than requiring exhaustive demonstrations of every configuration.

Inner Monologue (Huang et al., 2023) showed that providing LLMs with continuous environmental feedback during multi-step tasks dramatically improves success rates. For table setting, Inner Monologue uses object detection to verify each placement before proceeding to the next item, catching and correcting errors in real-time. This closed-loop approach achieved 91% success on 6-item casual settings versus 67% for open-loop execution, demonstrating the importance of verification feedback in long-horizon sequential tasks.

For the manipulation component, Diffusion Policy (Chi et al., 2023) has emerged as the preferred architecture for precise placement tasks. Diffusion Policy's ability to represent multi-modal action distributions is particularly valuable for table setting, where multiple valid placement trajectories exist (approach from left vs. right) but the final position must be precise. Training on 200-500 demonstrations per placement type (plate, fork, knife, glass), Diffusion Policy achieves sub-centimeter placement accuracy with 94% success rate on individual items.

Table setting data collection requires instrumented workstations with overhead and angled RGB-D cameras, a calibrated tabletop workspace, and a diverse inventory of real tableware. The primary sensor setup uses a high-resolution overhead camera (4K RGB at 30 Hz) for precise placement verification, supplemented by two angled side cameras for approach trajectory capture and a wrist-mounted camera for close-up grasp views. A depth sensor provides the 3D surface geometry needed for height-sensitive placement (stacking plates, positioning stemware).

Tableware diversity is critical for generalization. A production dataset should include at least 20 distinct place setting sets covering: standard ceramic dinnerware (various sizes and shapes), fine china (thin, fragile, with varying rim profiles), glassware (water tumblers, wine stems, champagne flutes), metal utensils (different handle profiles and weights), and cloth napkins (folded in multiple styles). Each setting should be demonstrated on at least 3 different table surfaces (bare wood, tablecloth, glass) to ensure the policy handles varying friction and reflectivity.

Annotation must capture both the absolute placement position and the relational structure of the setting. For each item placed, record: (1) final position in table coordinates (x, y, orientation, to millimeter precision), (2) relational predicates relative to neighboring items (left-of plate, above knife, aligned-with plate-center), (3) cultural compliance label (correct/incorrect per the specified setting style), and (4) placement quality score (centered, aligned, stable). These structured annotations enable training both end-to-end policies and hybrid systems that separate spatial reasoning from manipulation execution.

Claru collects table setting data using instrumented tabletop workstations with overhead 4K cameras, precision-calibrated table coordinate systems, and inventories of 20+ tableware sets spanning casual, formal, and culturally diverse conventions. Our operators demonstrate complete place settings following specified cultural rules, with each item placement annotated with sub-centimeter coordinates, relational predicates, compliance labels, and quality scores. We deliver datasets formatted for Diffusion Policy, ACT, and VLA architectures with train/validation/test splits stratified by tableware set and cultural style.