Video Prediction: Definition, Applications, and Training Data Requirements

Video prediction is the task of generating plausible future video frames given a sequence of past frames and optionally conditioning signals such as robot actions, language instructions, or goal images. The model takes as input a context window of observed frames and outputs predicted frames for a specified future horizon. The challenge is that the future is inherently uncertain — the same initial frames could lead to many different valid continuations depending on unobserved factors and stochastic events. Good video prediction models must capture both the deterministic aspects of scene dynamics (gravity, rigid body motion, kinematic constraints) and the stochastic aspects (which direction a person will walk, how objects will be rearranged).

In robotics, video prediction serves as a learned world model — an internal simulator that predicts the consequences of actions in the robot's actual visual environment. Classical model-based reinforcement learning uses explicit physics simulators as world models, but these simulators require precise knowledge of object geometries, masses, friction coefficients, and contact dynamics that are difficult to obtain in unstructured environments. Learned video prediction models bypass this requirement by predicting directly in pixel space: given what the camera currently sees and what action the robot plans to take, the model predicts what the camera will see after the action. This prediction-in-pixel-space approach works without explicit physical modeling because the training data implicitly encodes all relevant physics.

Action-conditioned video prediction adds a critical capability: the model predicts different futures depending on the action taken. This enables planning by generating predicted videos for multiple candidate actions and selecting the action whose predicted outcome best achieves the task goal. The planning loop is: observe the current scene, propose several candidate actions, generate predicted futures for each action using the video prediction model, evaluate each predicted future against the goal, and execute the best action. This model-predictive control approach has been demonstrated by UniPi, SuSIE, and AVID for tabletop manipulation tasks.

The training data requirements for video prediction models are substantial. The model must learn the visual appearance of diverse scenes (requiring millions of diverse video frames), the dynamics of physical interactions (requiring videos of objects being manipulated, falling, sliding, and colliding), and the mapping from actions to outcomes (requiring synchronized video-action pairs for the action-conditioned variant). Internet video provides visual diversity and general physical dynamics at scale, while robot-specific datasets provide the action-conditioned training signal. The most effective training recipes combine both sources, using internet video for pretraining and robot data for fine-tuning the action-conditioning mechanism.

Video prediction quality has improved dramatically with the advent of video diffusion models. These models generate video through iterative denoising, starting from noise and progressively refining it into a coherent video. The diffusion framework naturally handles the stochasticity of future prediction — different noise samples produce different valid futures from the same initial condition. Temporal attention mechanisms maintain consistency across generated frames, and classifier-free guidance enables trading off diversity for fidelity. Current models can generate physically plausible 2-4 second predictions at 256x256 resolution, sufficient for short-horizon robot planning.

For robotics teams, video prediction models are becoming practical tools for planning and data augmentation. The key practical decision is whether to use video prediction as an explicit planning module (generating predicted futures at runtime) or as an implicit world model whose knowledge is distilled into a policy during training.

Explicit planning with video prediction generates predicted videos at runtime and evaluates them against the goal. This approach, demonstrated by UniPi and SuSIE, is flexible — the same world model works for any goal specified in language or images — but computationally expensive. Generating a 2-second predicted video at 256x256 takes 1-5 seconds on an A100 GPU, and planning requires generating 10-100 candidate videos per action step. This makes explicit video planning practical for slow, deliberate manipulation but too slow for real-time reactive control.

Implicit world models train a policy that has internalized world knowledge without generating explicit video predictions at runtime. The training process uses video prediction as an auxiliary objective that forces the model to understand scene dynamics, but at deployment time the model directly outputs actions without generating frames. This approach is faster at runtime but less interpretable and less flexible — changing the goal requires retraining rather than re-specifying the goal to the planner.

Training data for video prediction should be diverse, high-quality, and capture realistic physical dynamics. Internet video provides visual diversity but often has editing cuts, camera motion unrelated to content, and compressed artifacts that degrade prediction quality. Egocentric video of human activities provides more robotics-relevant dynamics — hands manipulating objects in real environments — with continuous observation and natural physics. Claru's egocentric video catalog provides exactly this type of data: millions of frames of human-object interaction in diverse real environments, with the temporal continuity and physical realism that video prediction models need to learn accurate dynamics.

For action-conditioned video prediction, training data must include synchronized robot observations and actions. Claru can enrich egocentric video with pseudo-action labels (hand trajectories, grasp events, object displacements) derived from our vision model enrichment pipeline, providing a bridge between abundant human video data and the action-conditioned format that robotics world models require.

Claru provides the diverse, temporally coherent video data that video prediction models need to learn accurate physical dynamics. Our catalog of 3M+ annotated clips captures human-object interactions across thousands of real-world environments — kitchens, workshops, living spaces, outdoor settings — with the visual diversity, physical realism, and temporal continuity that video prediction models require.

For teams building world models for robot planning, Claru offers video data enriched with per-frame depth maps, optical flow, hand pose estimates, and object interaction labels. These enrichments provide additional supervision signals that improve video prediction quality: depth maps enable 3D-aware prediction, optical flow provides motion supervision, and object interaction labels help the model focus on physically meaningful dynamics rather than background motion. Our egocentric perspective matches the camera viewpoint of manipulation robots, providing directly relevant training data for action-conditioned video prediction in manipulation settings.