Visuomotor Policy: End-to-End Vision-to-Action Robot Control

A visuomotor policy is a neural network that maps raw visual observations (camera images) directly to robot motor commands (joint positions, end-effector poses, or torques), learning the complete perception-to-action chain as a single differentiable model. The term "visuomotor" combines "visual" (the input modality) with "motor" (the output modality), emphasizing the end-to-end nature of the mapping.

The end-to-end approach contrasts with the traditional modular pipeline in robotics, where separate modules handle perception (detecting objects and estimating their poses), planning (computing a trajectory from the current state to the goal), and control (converting trajectory waypoints to motor commands). Each module in the traditional pipeline introduces engineering effort, hand-tuned parameters, and error boundaries. A visuomotor policy replaces this entire pipeline with a single learned function, potentially capturing visual features and control strategies that a modular pipeline would miss.

Architecturally, a visuomotor policy consists of a vision encoder that processes camera images into a feature vector, and an action decoder that maps features to motor commands. The vision encoder is typically a convolutional neural network (ResNet) or Vision Transformer (ViT, DINOv2, SigLIP). The action decoder ranges from a simple MLP for single-step prediction to a Transformer with action chunking or a diffusion model for multi-step prediction. Modern visuomotor policies also incorporate proprioceptive state (joint positions, gripper width) as an additional input, concatenated with visual features before the action decoder.

The training signal for visuomotor policies comes from teleoperation demonstrations: paired recordings of what the robot's cameras saw and what actions the operator commanded at each timestep. The policy is trained through behavioral cloning (supervised learning on observation-action pairs), with the loss function being mean squared error for continuous actions or cross-entropy for discretized actions. The entire network — vision encoder and action decoder — is optimized end-to-end, allowing the vision encoder to learn features specifically useful for the control task rather than generic visual features.

Training visuomotor policies requires camera data and action labels with specific properties that affect the data collection and processing pipeline.

Camera calibration is essential. Visuomotor policies learn a mapping from pixel coordinates to physical actions, which depends on the camera's intrinsic parameters (focal length, principal point, distortion coefficients) and extrinsic parameters (position and orientation relative to the robot base). If the camera moves between data collection and deployment — even by a few millimeters — the policy may fail. Cameras should be rigidly mounted, and calibration should be verified before each collection session.

Image preprocessing affects policy performance more than many teams realize. The training images must match deployment conditions in resolution, aspect ratio, color space, and normalization. Common errors include training on center-cropped images but deploying with full-frame images, or training with one white-balance setting and deploying with another. Standardize image processing in a preprocessing function used identically during training and deployment.

The number of camera viewpoints affects both data requirements and policy capability. Each camera viewpoint requires its own vision encoder branch (or a shared encoder processing images sequentially), and all viewpoints must be present in every training demonstration. Adding a wrist camera to a scene camera approximately doubles the visual data bandwidth and increases training time, but provides close-up manipulation information that significantly improves grasp and insertion success rates. The tradeoff is worth it for contact-rich tasks.

For deployment, inference latency is a practical constraint. The policy must process camera images and output actions at the control frequency (10-20 Hz for manipulation). Large vision encoders (ViT-L with 304M parameters) run at approximately 10-15 Hz on an NVIDIA RTX 4090. Smaller encoders (ResNet-18, ViT-S) run at 30+ Hz but produce weaker representations. Quantization and TensorRT optimization can improve throughput by 2-3x without significant accuracy loss.

Claru provides the multi-viewpoint camera data and synchronized action labels that visuomotor policies require. Our teleoperation datasets include calibrated camera streams from wrist, head, and scene viewpoints, with all streams temporally synchronized to the action labels at the recording control frequency.

For teams training visuomotor policies, Claru ensures the visual diversity — varied lighting, backgrounds, object arrangements, and scene conditions — that produces robust policies. Each dataset delivery includes camera calibration files, recommended preprocessing parameters, and example dataloader code for PyTorch and JAX training pipelines.