Training Data for GENIMA (Generative Image as Action Models)

Claru provides the exact data modalities GENIMA requires for both training stages: time-synchronized RGB image streams, 7-DoF joint-position trajectories, gripper states, and calibrated camera parameters (intrinsics and extrinsics in JSON/YAML format). Our teleoperation data is collected on instrumented Franka Panda and UR5e platforms at 10-30 Hz with sub-millimeter joint-position accuracy from high-resolution encoders. For GENIMA's affordance image training, the critical requirement is accurate joint-position recordings that can be projected into the camera frame to render the colored sphere targets -- Claru's data includes full kinematic chain recordings and verified camera calibrations that make this projection straightforward. We also provide datasets with controlled visual diversity (varying objects, lighting, backgrounds) to validate GENIMA's canonical texture reversion property before deployment. Every dataset includes trajectory smoothness validation, success labeling, and format conversion to the GENIMA codebase's native HDF5 structure.

GENIMA (Generative Image as Action Models) is a behavior-cloning framework developed at the Dyson Robot Learning Lab by Mohit Shridhar, Yat Long Lo, and Stephen James. Published in July 2024 (arXiv 2407.07875), GENIMA introduces a fundamentally different approach to visuomotor policy learning: instead of regressing actions directly from images, it fine-tunes a pretrained image generation model to 'draw' target joint positions as colored spheres overlaid on the current RGB observation. A downstream ACT-style controller then converts those visual affordance targets into executable robot joint trajectories.

The key insight is that lifting the action representation into image space lets the policy inherit the strong visual priors of internet-pretrained diffusion models. Because Stable Diffusion already understands geometry, lighting, and object identity, the resulting policy is far more robust to visual perturbations than a standard visuomotor approach. GENIMA outperforms ACT in 16 out of 25 RLBench simulation tasks and beats Diffusion Policy in all 25 tasks, while also demonstrating strong generalization on 9 real-world manipulation tasks with a Franka Panda arm.

A distinctive emergent property of GENIMA is canonical texture reversion: the diffusion model tends to normalize object appearances to canonical colors and textures during affordance image generation, making the policy invariant to randomized object colors, distractors, lighting changes, and background textures. This property is not explicitly trained but arises from the visual priors of the pretrained diffusion backbone.

GENIMA's architecture is a two-stage pipeline. Stage 1 is an affordance image generator built by fine-tuning SD-Turbo with ControlNet. The ControlNet adapter takes the current RGB observation as a spatial conditioning signal, while the text encoder receives the language instruction. The denoising UNet then generates an output image identical to the observation but with colored spheres overlaid at the predicted 3D positions of each robot joint, projected into the camera frame. Because SD-Turbo is a single-step distilled model, this generation requires only one forward pass, keeping inference latency low.

Stage 2 is an ACT-based controller that consumes the generated affordance image. The controller uses a ResNet-18 backbone to encode the affordance image and a Transformer decoder to predict a chunk of future joint-position actions. The ACT architecture enables temporal action chunking, producing smooth multi-step trajectories from a single affordance image rather than requiring per-timestep replanning.

The two-stage decomposition offers a clean separation of concerns. The diffusion model handles the 'what' and 'where' of manipulation (spatial reasoning, object identity, grasp point selection), while the ACT controller handles the 'how' (motion planning, trajectory smoothness, dynamics). This separation means each stage can be trained and debugged independently, and the affordance images provide an interpretable intermediate representation that engineers can visually inspect.

A critical innovation is the use of internet-pretrained visual priors for policy robustness. Because the diffusion backbone was trained on billions of internet images, it implicitly encodes strong priors about 3D geometry, lighting, material properties, and object categories. These priors transfer directly to the manipulation policy, enabling zero-shot generalization to novel object instances, colors, and backgrounds that would require explicit domain randomization in conventional approaches.

GENIMA's data-efficiency is one of its most compelling properties. In the RLBench evaluation, each of the 25 tasks was trained with as few as 20 to 50 demonstrations, a fraction of what standard visuomotor policies require. Each demonstration consists of an RGB observation stream (128x128 resolution in simulation, 480x640 in real-world), paired with ground-truth robot joint positions at each timestep. From these joint positions, the training pipeline renders colored sphere overlays on the observation images to create the affordance image targets used to fine-tune the diffusion model.

For real-world deployment, the authors collected 9 manipulation tasks on a Franka Panda arm with a wrist-mounted RealSense camera. Each task used between 20 and 50 teleoperated demonstrations. The teleoperation data includes RGB frames, 7-DoF joint positions, and gripper state at 10 Hz. The affordance sphere rendering is computed offline from the recorded joint trajectories using known camera intrinsics and extrinsics, so no special annotation beyond standard teleoperation recording is needed.

The ControlNet fine-tuning of SD-Turbo requires paired (observation, affordance_image) samples. The observation is the raw RGB frame, and the affordance image is the same frame with colored spheres at each joint position projected into the camera view. Training typically converges in 10,000 to 50,000 gradient steps with a batch size of 8 to 16 on a single A100 GPU. The ACT controller is trained separately on the affordance images and corresponding action chunks, using the same demonstration dataset.

For teams looking to extend GENIMA to new tasks, the minimum viable dataset is approximately 20 high-quality teleoperated demonstrations per task, with consistent camera placement and calibrated intrinsics. Increasing to 50 demonstrations per task provides meaningful improvements in success rate, particularly for tasks involving articulated objects or precise contact dynamics.

Claru provides teleoperated robot demonstration datasets with the precise data modalities GENIMA requires: time-synchronized RGB image streams, 7-DoF joint positions, gripper states, and calibrated camera parameters. Our collection pipeline uses instrumented Franka Panda and UR5e setups with RealSense and ZED cameras, producing data at 10-30 Hz with sub-millimeter joint-position accuracy from high-resolution encoders.

For GENIMA specifically, the critical data annotation is the joint-position trajectory at each timestep, which is used to render the affordance sphere targets. Claru's data includes full kinematic chain recordings (joint positions, end-effector poses, and gripper width) that can be directly projected into the camera frame to generate GENIMA's colored sphere training targets. We also provide camera intrinsic and extrinsic calibration files in standard formats (JSON, YAML) compatible with the GENIMA codebase.

Beyond raw demonstrations, Claru can supply datasets with controlled visual diversity -- varying object instances, lighting conditions, backgrounds, and distractor objects -- that stress-test GENIMA's canonical texture reversion property and verify generalization before deployment. Our data quality pipeline includes trajectory smoothness validation, camera calibration verification, and demonstration success labeling to ensure that every sample in the training set represents a successful task completion.