Something-Something V2 Alternative: Robot-Specific Training Data for Physical AI

Something-Something V2 occupies a unique position in the robotics data landscape: it is not a robotics dataset at all, but its focus on temporal object manipulation dynamics has made it a popular pretraining resource for robotic visual encoders. Research from R3M, Voltron, and related work demonstrated that SSv2-pretrained representations outperform ImageNet and Kinetics representations for manipulation tasks, precisely because SSv2's content involves physical object interactions rather than scene recognition. However, SSv2's utility for robotics is strictly limited to visual pretraining. It provides no robot-specific observations, no action trajectories, no multi-modal sensor data, and crucially, no commercial license. Claru provides everything SSv2 cannot: expert robot demonstrations with full action trajectories on your specific platform, multi-modal observations including depth and force/torque, calibrated camera geometry, and natural language annotations -- all under a commercial license suitable for product deployment. For teams that have used SSv2 to bootstrap visual representations, Claru delivers the robot-specific data needed to convert those representations into deployed manipulation policies. As robot-specific pretraining corpora like OXE grow, Claru's demonstrations serve double duty: both as fine-tuning data for SSv2-pretrained encoders and as part of the robot-specific pretraining mix that is replacing human video pretraining in the field.

Something-Something V2 (SSv2) is a large-scale video understanding dataset created by TwentyBN (later acquired by Qualcomm AI Research) and published in 2017 with a major V2 update. It contains 220,847 short video clips showing crowd-sourced workers performing 174 predefined actions with everyday objects. Unlike datasets that rely on appearance cues (a beach suggests 'surfing'), SSv2 was specifically designed to require temporal reasoning -- understanding the sequence and dynamics of an action, not just recognizing a static scene.

Each video shows a human hand interacting with an object on a surface: pushing something left to right, picking something up, turning something upside down, covering something, folding something, pretending to put something on a surface, and 168 other actions. The actions are defined as templates with object slots (e.g., 'Pushing [something] from left to right'), where the object varies across clips. This template structure means models must learn the action dynamics (the 'pushing' motion) rather than memorizing object-action correlations.

SSv2 became a standard benchmark for video understanding models and has been influential in robotics research as a pretraining dataset. Because the actions involve physical object manipulation with temporal structure (beginning, middle, end of an action), SSv2 representations capture manipulation-relevant visual dynamics that ImageNet or Kinetics pretraining does not. Researchers have used SSv2-pretrained video encoders as visual backbones for robotic manipulation policies, including early VLA work.

The dataset is released under a research-only license controlled by the TwentyBN terms of use, which does not permit commercial use. This is a critical distinction from many robotics datasets. SSv2 provides only RGB video and action class labels -- no depth, no 3D pose, no force measurements, and no robot-specific observations.

The fundamental limitation of SSv2 for robotics is the embodiment gap. SSv2 shows human hands, not robot grippers. Human hands have 27 degrees of freedom, compliant fingers, tactile sensitivity, and dexterous grasp capabilities that no current robot gripper matches. A model pretrained on SSv2 learns the visual dynamics of human manipulation, which differ fundamentally from robot manipulation: different grasp strategies, different approach angles, different contact patterns, and different failure modes. The visual representations may transfer partially, but they carry assumptions about manipulation physics that do not hold for robot end-effectors.

SSv2 contains no actionable robot data. There are no joint positions, end-effector poses, gripper states, or motor commands -- the data modalities that policies consume to produce robot behavior. SSv2 can only serve as a visual pretraining source (learning representations), never as direct training data for a manipulation policy. The jump from 'recognizing that a human is pushing something' to 'controlling a robot to push something' requires entirely different data.

The research-only license is a hard blocker for commercial deployment. The TwentyBN terms of use explicitly restrict commercial use. Any model pretrained on SSv2 that is deployed in a commercial product carries licensing risk. This is not a gray area -- the terms are clear. Teams building products need pretraining data with commercial rights, which SSv2 does not provide.

SSv2's 174 action categories are coarse, classification-oriented labels designed for video understanding benchmarks. They describe what happens ('pushing left to right') but not how -- the fine-grained trajectory, the force profile, the grip configuration, the timing of contact transitions. Robot policies need the 'how' at a control-frequency level, not just the 'what' at a video-clip level.

Camera viewpoints in SSv2 are uncontrolled -- each crowd worker used their own phone or webcam at whatever angle was convenient. This creates viewpoint variability that is useful for robust visual recognition but provides no calibrated spatial information. Robot manipulation requires known camera geometry (intrinsics, extrinsics) to reconstruct 3D spatial relationships between the robot, objects, and environment.

SSv2 is valuable for academic research on temporal video understanding, action recognition, and video representation learning. If your research involves developing or benchmarking video models that must understand the temporal dynamics of object manipulation (not just static scene recognition), SSv2's template-based action categories provide the right evaluation framework.

SSv2 also has documented value as a pretraining dataset for robotic visual encoders in research settings. Multiple papers have shown that video encoders pretrained on SSv2 produce better visual features for manipulation than those pretrained on Kinetics or ImageNet, because SSv2's content is closer to the physical interactions robots perform. This pretraining benefit is real but diminishes as more robot-specific pretraining corpora (like OXE) become available.

Switch to Claru for any path that leads to commercial deployment. If you need data to train or fine-tune a manipulation policy, if you need commercial licensing, if you need robot-specific observations, or if you need action trajectories rather than classification labels -- Claru provides data that SSv2 structurally cannot. Our demonstrations are collected on real robots performing real tasks, with the full observation and action space your policy requires.

For teams that have leveraged SSv2 for visual encoder pretraining, Claru provides the robot-specific data needed to train a complete manipulation policy. SSv2 teaches your visual backbone about the temporal dynamics of physical interactions; Claru teaches your policy how your specific robot should execute those interactions, with full action trajectories, multi-modal observations, and task-specific language annotations.

Claru addresses SSv2's embodiment gap by collecting data on your actual robot. Where SSv2 shows a human hand pushing a mug, Claru shows your gripper pushing your target objects in your environment. The visual features, grasp configurations, and contact dynamics are specific to your hardware, eliminating the need for cross-embodiment transfer between human hands and robot end-effectors.

Our commercial license removes the legal risk that SSv2's research-only terms create. Models trained on Claru data can be deployed in commercial products without licensing concerns. For teams that have been using SSv2 in research and are now commercializing their approach, Claru provides a clean licensing path forward.

Data is delivered with the full multi-modal sensor suite that SSv2 lacks: synchronized RGB-D from calibrated cameras, proprioception, force/torque, and optional tactile data, alongside standardized action labels at your control frequency. This transforms a visual pretraining resource into a complete policy training pipeline.