Real-World Data for ARNOLD

AI2-THOR photorealistic rendering reduces visual gap but physical interactions (pouring, heating) use simplified state transitions rather than continuous physics. Language diversity in evaluation is limited to template-based variations.

ARNOLD represents the convergence of natural language processing and robotic manipulation benchmarking. The benchmark tests whether models can understand language instructions, ground them in visual observations, and execute the corresponding physical manipulation — the core capability needed for language-conditioned robots.

The language grounding challenge in ARNOLD extends beyond simple object references. Instructions like 'pour the coffee into the blue mug on the counter' require resolving spatial references, understanding object attributes, and decomposing the instruction into a sequence of physical actions. Current VLA models achieve moderate success on template-based instructions but struggle with the compositional complexity of natural language.

AI2-THOR's photorealistic rendering narrows the visual domain gap compared to MuJoCo or PyBullet-based benchmarks, but the physical interaction gap remains wide. Pouring in AI2-THOR is a state transition (liquid teleports from container A to container B), not a physical simulation of fluid dynamics. Similarly, opening doors and drawers uses simplified hinge models. Real-world data where these physical interactions actually occur provides the ground truth for evaluating whether models understand physical causality or just memorize simulation-specific shortcuts.

The instruction diversity gap is particularly important. Template-based evaluation ('pick up the [object] from the [location]') tests a narrow slice of language understanding. Real humans give instructions in diverse ways — elliptical, indirect, contextual. Training and evaluation on real language instructions, paired with actual demonstrations, tests genuine language grounding rather than template matching.