Training Data for PaLM-E

Claru provides the embodied interaction data that PaLM-E-style models require: RGB observation sequences paired with natural language task descriptions and step-by-step execution plans. Our annotators produce planning-level descriptions ('pick up the blue mug, move to the sink, place mug in sink') rather than motor-level labels, matching PaLM-E's training format. We deliver data as interleaved image-text episodes in JSON format compatible with open-source VLM fine-tuning pipelines. Our collection covers kitchen manipulation, tabletop rearrangement, mobile manipulation, and multi-object sequential tasks -- the task domains PaLM-E was designed to handle.

PaLM-E (Pathways Language Model - Embodied) is an embodied multimodal large language model introduced by Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, and collaborators at Google DeepMind and TU Berlin in March 2023 (arXiv:2303.03378). At 562 billion parameters, PaLM-E is the largest vision-language model ever applied to robotics. Its core innovation is embedding continuous sensor data -- RGB images, scene state vectors, and object-centric representations -- directly into the token sequence of a large language model (PaLM 540B), enabling the LLM to reason about the physical world and generate executable robot plans.

The key insight behind PaLM-E is that robot planning requires grounded reasoning: understanding what objects are in a scene, what actions are physically possible, and how those actions change the state of the world. Traditional robot planners rely on hand-crafted symbolic representations of the world, which break down when the environment is complex or novel. PaLM-E replaces this symbolic pipeline by training an LLM to reason directly from raw sensor data, generating step-by-step plans in natural language that a low-level controller then executes.

PaLM-E demonstrated several breakthrough capabilities. It could perform multi-step mobile manipulation planning from a single RGB image ('bring me the chips from the kitchen counter'), reason about affordances ('can I push the heavy box?' vs. 'can I push the paper cup?'), handle failures by re-planning when an action does not produce the expected outcome, and transfer knowledge across robot embodiments. Crucially, Driess et al. showed that PaLM-E exhibited positive transfer: training on a mixture of internet-scale vision-language data and robot-specific interaction data produced a model that was better at both tasks than models trained on either data source alone.

PaLM-E operates at the planning level, not the motor control level. It generates high-level action plans ('1. Navigate to counter. 2. Pick up bag of chips. 3. Navigate to user. 4. Hand over chips.') that are then executed by lower-level policies (e.g., RT-1 or SayCan primitives). This architectural separation means PaLM-E's data requirements are fundamentally different from end-to-end VLA models like OpenVLA or Octo: instead of continuous motor commands, PaLM-E consumes and produces natural language plans grounded in visual observations.

PaLM-E's architecture is a standard decoder-only transformer (PaLM 540B) with one critical modification: continuous sensor inputs are projected into the language model's token embedding space and interleaved with text tokens in the input sequence. For visual inputs, ViT-22B encodes an RGB image into a sequence of patch embeddings (e.g., 256 tokens for a 224x224 image with 14x14 patches). These visual tokens are then linearly projected into the same dimensionality as PaLM's text token embeddings and inserted at specific positions in the input sequence marked by special placeholder tokens.

The paper explored three types of sensor encodings: (1) ViT-based image encoding, where raw RGB images are processed through a Vision Transformer to produce patch-level embeddings; (2) object-centric representations using OSRT (Object Scene Representation Transformer), which encodes the scene as a set of object-level feature vectors; and (3) state vector encoding, where low-dimensional scene state (object positions, orientations, bounding boxes) is linearly projected into the token space. The ViT-based encoding performed best overall, but combining ViT features with state vectors showed the strongest results on tasks requiring precise spatial reasoning.

A major innovation in PaLM-E is the demonstration of positive transfer between internet-scale VLM training and robot-specific training. When trained jointly on visual question answering (VQA), image captioning, and robot planning tasks, PaLM-E performed better on all tasks compared to models trained on any single data mixture. This transfer was especially pronounced at scale: the 562B model showed stronger positive transfer than the 12B and 84B variants. This finding suggests that the knowledge encoded in web-scale language and vision data (commonsense reasoning, object recognition, spatial understanding) directly benefits embodied reasoning.

PaLM-E also introduced geometric input encodings that inject 3D structural information into the language model. Neural radiance fields (NeRFs) and point clouds can be encoded as token sequences and inserted alongside image and text tokens. This allows PaLM-E to reason about 3D spatial relationships, occlusions, and viewpoint-dependent visibility -- capabilities that are critical for real-world robot planning but absent from most 2D vision-language models.

PaLM-E's training data is a mixture of internet-scale vision-language data and robot-specific interaction data. The internet-scale component is standard VLM training data: image-text pairs from web crawls, visual question answering datasets (VQAv2, OK-VQA), image captioning datasets (COCO Captions), and other multimodal benchmarks. This data teaches the model visual recognition, commonsense reasoning, and language fluency. The paper reports training on the same data mixture as PaLI (Chen et al., 2022), which includes billions of image-text pairs.

The robot-specific component consists of embodied interaction data from three platforms: (1) a mobile manipulation robot (similar to the Everyday Robots platform) performing kitchen tasks like fetching objects, opening drawers, and wiping surfaces; (2) a tabletop robot arm performing language-conditioned pick-and-place and stacking tasks; and (3) a simulated tabletop environment (Language-Table) with pushing and arrangement tasks. Each robot interaction episode contains: an initial RGB observation, a natural language task description, a sequence of intermediate observations, and the step-by-step plan (in natural language) that was executed to accomplish the task.

The critical data format for robot interaction episodes is the interleaved multi-modal sequence. A training example looks like: [image_1] 'Bring me the chips from the counter.' [image_2] 'Step 1: Navigate to counter.' [image_3] 'Step 2: Pick up bag of chips.' [image_4] 'Step 3: Navigate to user.' [image_5] 'Step 4: Hand over chips.' -- where each [image_N] is an RGB frame from the robot's camera encoded as visual tokens. The model learns to generate the step-by-step text plan conditioned on the interleaved visual observations.

For teams building PaLM-E-style embodied reasoning systems, the key data requirement is paired visual observation + language plan data. Each interaction episode needs: (1) RGB images captured at key moments during task execution (typically 3-10 frames per episode showing the start state, intermediate states, and goal state); (2) a natural language task description; and (3) a step-by-step natural language plan describing the actions taken. Annotations should describe what the robot does in each step at a semantic level ('pick up the blue cup'), not at a motor level ('move end-effector to [0.3, 0.2, 0.1]'). This planning-level annotation is what distinguishes PaLM-E data from the motor-level data required by VLA models like OpenVLA or Octo.

Claru provides the embodied interaction data that PaLM-E-style models require: RGB observation sequences paired with natural language task descriptions and step-by-step execution plans. Our annotation pipeline produces planning-level descriptions (not motor-level action labels) -- human annotators watch robot task execution videos and write the multi-step plans that a language model should learn to generate. Each episode includes 3-10 keyframe observations with paired plan steps, matching the interleaved multi-modal format PaLM-E uses.

For teams building PaLM-E-inspired systems at smaller scale (using open-source LLM backbones like Llama 3 or Gemma 2 instead of the closed PaLM 540B), Claru delivers data in a flexible format: JSON episodes with base64-encoded or file-referenced images, task descriptions, and step-by-step plans that can be tokenized and interleaved by any VLM training pipeline. We also provide scene-level metadata -- object identities, approximate positions, and affordance labels -- that can be used as supplementary state vector inputs alongside RGB observations.

Claru's collection network spans the task domains PaLM-E was designed to handle: kitchen manipulation, tabletop rearrangement, mobile manipulation in home and office environments, and multi-object interaction tasks requiring sequential planning. We capture both RGB keyframes and full video for each episode, enabling teams to choose their preferred observation sampling strategy. All language annotations are written by trained annotators following a planning-level description protocol that produces the grounded, actionable text PaLM-E's training objective expects.