Optical Flow: Per-Pixel Motion Estimation for Video Understanding and Robotics

Optical flow is the pattern of apparent motion of visual elements in a scene between two consecutive observations, represented as a dense 2D vector field. For each pixel (x, y) in a reference frame I_t, the optical flow field (u, v) specifies the displacement to the corresponding pixel location (x+u, y+v) in the subsequent frame I_{t+1}. The resulting flow field has the same spatial dimensions as the input images but two channels (horizontal and vertical displacement), encoding where every visible surface point moved between frames.

The computation rests on the brightness constancy assumption: a pixel's intensity does not change between frames, only its position. Formally, I(x, y, t) = I(x+u, y+v, t+1). This assumption fails under lighting changes, specular reflections, transparency, and large displacements, making robust flow estimation a long-standing challenge. Modern deep learning methods relax this assumption by learning motion patterns from data, handling the violations through implicit priors captured during training.

Optical flow captures two sources of motion: ego-motion (the camera or robot moving) and independent object motion (things in the scene moving). Decomposing total flow into these components is essential for robotics. Ego-motion flow is geometrically constrained — it can be predicted from the camera's rotation and translation plus the scene's depth — while object motion flow is unconstrained and signals dynamic elements the robot must react to. Scene flow extends optical flow to 3D, estimating per-point 3D motion vectors rather than 2D image displacements, directly providing the information needed for robotic spatial reasoning.

For robotics training data, optical flow serves both as a target for training flow estimation models and as an intermediate representation for other tasks. Video prediction models use flow to warp frames forward in time. Action recognition models use flow as a motion feature channel. Video object segmentation uses flow to propagate annotations across frames. The quality and coverage of flow training data — particularly for fast motions, thin structures, and occlusion boundaries — directly determines the reliability of all downstream applications that consume flow estimates.

For robotics teams using optical flow, key practical decisions involve model selection, runtime optimization, and integration with the perception stack. RAFT with 12 iterations achieves the best accuracy but runs at only 10-15 FPS on an NVIDIA Jetson Orin at 640x480 resolution. Reducing to 6 iterations roughly doubles speed with modest accuracy degradation. RAFT-Small runs at 40+ FPS with approximately 15% higher endpoint error. For real-time navigation at 30+ FPS, lightweight models or hardware-accelerated Lucas-Kanade tracking may be necessary, with RAFT reserved for offline analysis or lower-rate dense flow computation.

Training data strategy depends on whether you need a general-purpose flow model or a domain-specific one. General-purpose models pretrained on FlyingChairs and FlyingThings3D transfer well to most real-world scenes. For robotics-specific improvements, fine-tuning on synthetic data rendered from the target environment (using simulators like Isaac Sim or MuJoCo with domain randomization) typically improves accuracy by 10-20% on in-domain evaluation. Real-world flow ground truth is extremely expensive to obtain (requiring multi-view rigs and 3D reconstruction), so synthetic data remains the primary training source.

Integration patterns in robotics stacks vary by application. For visual odometry, flow is decomposed into rotational and translational components using the camera intrinsic matrix and the essential matrix. For object segmentation, flow inconsistent with the ego-motion model signals independently moving objects. For manipulation feedback, flow in the gripper region indicates whether the object is being successfully manipulated. For video annotation pipelines, flow-based mask warping accelerates temporal annotation by propagating labels from keyframes to intermediate frames with human correction at occlusion boundaries.

Claru provides high-frame-rate video datasets optimized for optical flow computation and downstream motion understanding tasks. Our egocentric and third-person video datasets are captured at 30 FPS with controlled exposure settings that minimize motion blur — the primary failure mode for flow estimation in robotics scenarios. For teams training custom flow models, Claru provides video from indoor robotics environments (kitchens, warehouses, workspaces) with the fast hand motions, close-range object interactions, and dynamic lighting that standard synthetic benchmarks do not cover. We also use optical flow as a core tool in our annotation pipeline: RAFT-based flow computation enables mask propagation across video frames, reducing per-frame segmentation annotation time by 3-5x. With 386,000+ video clips in our catalog spanning diverse environments and activity types, Claru provides the temporal data density that motion understanding models require.