Training Data for RoboFlamingo

Claru provides VLM-compatible demonstration data for RoboFlamingo-style training: multi-frame RGB observations from static and gripper cameras at 5 Hz, paired natural language instructions with 3-5 paraphrases per task type, and continuous 7-DoF action labels. We deliver data in HDF5 or RLDS format with observation history windows pre-constructed for direct ingestion into Flamingo cross-attention pipelines. For long-horizon evaluation, we provide task chaining sequences (3-5 subtasks per sequence) matching the CALVIN evaluation protocol. Our typical delivery is 500-5,000 demonstrations across 10-30 tasks with camera calibration metadata and automated success classifiers.

RoboFlamingo is a vision-language-action model developed by Xinghang Li, Minghuan Liu, Hanbo Zhang, and collaborators at ByteDance Research, published in 2023 (arXiv:2311.01378). It adapts DeepMind's Flamingo vision-language model -- originally designed for visual question answering and image captioning -- for robot manipulation by adding a lightweight action prediction head on top of the frozen (or partially unfrozen) VLM backbone. The central result: RoboFlamingo achieved state-of-the-art performance on the CALVIN long-horizon manipulation benchmark, completing 4.2 out of 5 chained subtasks in sequence -- surpassing all prior methods at the time of publication.

The key insight behind RoboFlamingo is that pre-trained vision-language models already encode rich visual understanding, spatial reasoning, and language grounding from web-scale pre-training. Rather than training a robot policy from scratch (which requires massive amounts of robot data), RoboFlamingo reuses the Flamingo VLM's representations and only trains a small action head to map those representations to robot actions. This transfer learning approach dramatically reduces the amount of robot-specific data needed: RoboFlamingo achieves strong performance with as few as a few hundred demonstrations per task.

RoboFlamingo processes a history of RGB observations (multiple frames from one or more cameras) and a natural language instruction through Flamingo's cross-attention-based architecture, where visual features from a frozen CLIP ViT encoder are attended to by the language model via gated cross-attention layers. The language model (based on Chinchilla/LLaMA-style architectures) produces a contextualized representation that captures both what is in the scene and what the instruction asks for. A lightweight MLP action head then predicts continuous end-effector actions from this representation.

RoboFlamingo's position in the robot learning landscape is as a demonstration of efficient VLM-to-robot transfer. While later models like OpenVLA and pi-zero have surpassed it in scale and capability, RoboFlamingo was one of the first to show that a frozen VLM backbone with a trained action head can compete with or beat end-to-end trained robot policies. This finding has influenced the design of nearly every subsequent VLA model.

RoboFlamingo's architecture has three components. First, the visual encoder: a frozen CLIP ViT-L/14 processes each RGB observation frame independently, producing a sequence of visual tokens (patch embeddings) for each frame. The visual tokens from multiple frames (the observation history window) are all available for cross-attention. Second, the language model: a Flamingo-style decoder processes the language instruction tokens while attending to the visual tokens via gated cross-attention layers. These layers allow the language model to selectively attend to relevant visual information at each layer of the transformer, building up a representation that fuses language and vision. Third, the action head: a lightweight MLP (typically 2-3 layers, approximately 10M parameters) maps the language model's final hidden state to a 7-DoF continuous action prediction.

The gated cross-attention mechanism is the key architectural element inherited from Flamingo. In each transformer layer, after self-attention on the text tokens, a cross-attention layer attends from text tokens to visual tokens. A learnable gating parameter (initialized to zero) controls how much visual information flows into the language model at each layer. This gating mechanism allows the pre-trained language model to be gradually and stably fine-tuned to incorporate visual information without catastrophic forgetting of its language capabilities.

A critical design decision in RoboFlamingo is how much of the VLM backbone to unfreeze during robot fine-tuning. The paper evaluates three strategies: (1) fully frozen backbone with only the action head trained -- fastest and most data-efficient but limited in representational adaptation; (2) unfreezing the top N transformer layers while keeping the visual encoder and lower layers frozen -- the recommended approach, balancing adaptation and stability; (3) full fine-tuning of all parameters -- requires the most data and risks overfitting on small datasets. The paper found that unfreezing the top 2-4 transformer layers produced the best CALVIN benchmark results.

RoboFlamingo also introduced the use of observation history for VLM-based robot policies. Rather than processing a single observation frame (as in RT-2 or the original VLA concept), RoboFlamingo feeds 6-12 recent frames to the visual encoder, giving the language model temporal context. This is critical for tasks that require understanding motion direction, tracking objects across time, or detecting whether an action has succeeded. The multi-frame approach leverages Flamingo's native ability to process interleaved image-text sequences, which was originally designed for few-shot image understanding from multiple examples.

RoboFlamingo's data requirements depend on which components are being trained. For the action head only (frozen VLM backbone), a few hundred demonstrations per task can produce strong results. The CALVIN benchmark evaluation used approximately 24,000 demonstrations across 34 tasks in a simulated tabletop environment, but the paper showed that strong performance was achievable with subsets of this data. For real-world fine-tuning with a partially unfrozen backbone, 200-1,000 demonstrations per task is a practical range.

Each demonstration consists of a sequence of timesteps, where each timestep contains: an RGB observation (200x200 for CALVIN, typically 224x224 or higher for real-world), a 7-DoF action vector (position delta, orientation delta, gripper command), and a natural language instruction describing the task. The observation history window means each training sample actually includes 6-12 consecutive frames, not just a single frame. Actions should be recorded at the control frequency (5 Hz for standard RoboFlamingo).

Language instructions are essential for RoboFlamingo's performance. The Flamingo backbone's language understanding is what enables task conditioning and generalization to novel task descriptions. Instructions should be natural language descriptions of the task ('push the red block to the left,' 'open the drawer and pick up the pink block'), not symbolic codes. Instruction diversity matters: using varied paraphrases across demonstrations of the same task improves generalization. The CALVIN benchmark provides 34 distinct task types with natural language descriptions, and RoboFlamingo chains up to 5 of these tasks in sequence.

For real-world deployment, the main data quality considerations are: (1) camera consistency -- the visual encoder (CLIP ViT) is sensitive to camera placement changes, so camera positions should be fixed within each task environment; (2) temporal consistency -- observations should be captured at a steady 5 Hz with no dropped frames or timestamp irregularities; (3) action quality -- demonstrations should be smooth and successful, with jerky or failed episodes filtered out; and (4) multi-view support -- RoboFlamingo can process observations from multiple cameras (e.g., static overhead + gripper-mounted), which improves performance on fine manipulation tasks.

Claru provides VLM-compatible demonstration data for RoboFlamingo-style training pipelines. Our datasets include multi-frame RGB observation histories (6-12 frames per sample at 5 Hz), paired natural language task instructions with varied paraphrases, and continuous 7-DoF end-effector action labels. Every demonstration is captured from fixed camera positions with consistent lighting conditions, matching the stability requirements of CLIP ViT-based visual encoders.

For teams building RoboFlamingo-style models for real-world tasks, Claru provides demonstrations from both static workspace cameras and gripper-mounted cameras, synchronized at 5 Hz. Our annotation team writes natural language instructions for each demonstration, with 3-5 paraphrases per task type to build the instruction diversity that improves Flamingo-backbone generalization. We also provide task chaining annotations for long-horizon evaluation: sequences of 3-5 subtasks that test the model's ability to complete multi-step manipulation plans.

Claru's typical delivery for a RoboFlamingo project is 500-5,000 demonstrations across 10-30 task types, with multi-camera observations, diverse language annotations, and pre-defined evaluation sequences. We provide data in HDF5 or RLDS format with all fields the training pipeline expects: observation tensors, action vectors, language strings, episode boundaries, and camera calibration metadata. For teams evaluating on CALVIN-style benchmarks, we also provide task success classifiers trained on our data to enable automated evaluation.