Pushing Task Training Data

Claru provides pushing data collection for both systematic push physics research and goal-directed manipulation applications. Our stations feature overhead RGB-D cameras with AprilTag fiducial tracking at sub-millimeter accuracy, calibrated planar surfaces with characterized friction coefficients, and robot arms with fingertip force sensors for push contact measurement. We support systematic parameter-sweep collection (100-500 pushes per object per hour) and teleoperated goal-directed demonstrations for singulation, sorting, and arrangement tasks. Deliverables include pre/post-push object poses, push parameter annotations, contact force profiles, and per-push state labels — formatted for push prediction networks, Diffusion Policy, RL environments, or custom architectures. Our throughput enables scaling from research datasets (10K-50K pushes) to production-scale corpora (200K+ interactions) on real client objects.

Pushing — moving objects by applying forces through sustained contact without grasping — is one of the most fundamental yet underappreciated manipulation skills. Humans push objects constantly: sliding a plate across a table, nudging a box into alignment, sweeping crumbs into a dustpan. For robots, pushing is essential when objects are too large, too flat, or too heavy to grasp, and as a pre-manipulation strategy to singulate cluttered objects or reposition items into graspable configurations. Pushing is also the simplest form of contact-rich manipulation, making it an ideal testbed for learning physics-based manipulation policies.

The physics of pushing are deceptively complex. When a finger pushes an object on a surface, the resulting motion depends on the push point relative to the center of friction, the friction coefficient between the object and surface, the object's mass distribution, and the contact geometry between the finger and object. Lynch and Mason (1996) formalized the mechanics of planar pushing, showing that the push-to-slide transition depends on the ratio of pushing force to normal load and the eccentricity of the push point. A push applied through the center of friction produces pure translation, while eccentric pushes produce coupled translation and rotation — a behavior that is intuitive for humans but requires explicit physics reasoning for robots.

The MIT Push Dataset (Yu et al., 2016) was the seminal contribution that enabled data-driven push prediction. Containing 250,000 push interactions across 11 object shapes with systematic variation of push parameters (location, angle, velocity), this dataset showed that neural network push predictors trained on real data outperform analytical models by 40% on held-out objects because they implicitly learn material-specific friction behaviors that analytical models must approximate. This result established that real-world push data is essential for accurate push prediction.

Modern applications of push manipulation extend well beyond research benchmarks. In warehouse automation, push-based singulation separates touching objects in cluttered bins to enable individual grasping — Amazon reports that singulation pushes are required for 30-40% of bin picks in typical e-commerce inventory. In manufacturing, push-based fixture loading aligns parts against reference surfaces with sub-millimeter precision. In household robotics, pushing is the primary strategy for clearing surfaces, arranging objects, and operating sliding mechanisms (drawers, doors). Building robust push policies for these diverse applications requires demonstration data that captures the full range of object-surface friction interactions.

The MIT Push Dataset (Yu et al., 2016) established the foundational benchmark for data-driven push prediction. Training a neural network on 250,000 push interactions to predict the 3-DoF outcome (delta x, delta y, delta theta), the resulting model achieves 2.3 mm average position error and 1.7-degree average rotation error on held-out pushes — 40% better than the quasi-static analytical model. Importantly, the learned model generalizes to novel objects with only 50-100 calibration pushes per new object, compared to thousands of pushes needed to fit analytical model parameters.

PushNet (Li et al., 2018) extended push prediction to visual observations, training an encoder-decoder network that takes an overhead RGB image and push parameters as input and predicts the post-push object configuration. On the MIT Push Dataset benchmark, PushNet achieves 3.1 mm position error from images alone — only 35% worse than models with ground-truth pose access. This demonstrated that visual push prediction is viable for real-world deployment where precise object pose estimation may not be available.

For multi-step push manipulation, Diffusion Policy (Chi et al., 2023) achieved breakthrough results on the PushT benchmark — a task requiring pushing a T-shaped block to a target configuration. Diffusion Policy achieves 88.2% success rate compared to 72.3% for BC-RNN and 65.1% for IBC, establishing the state of the art for goal-conditioned push manipulation. The advantage comes from modeling the multimodal action distribution inherent in pushing — there are many valid push sequences to reach the same goal, and Diffusion Policy generates diverse, high-quality solutions while other methods collapse to suboptimal averages.

Recent work on foundation models for pushing has shown promising generalization. RT-2 (Brohan et al., 2023) demonstrates zero-shot push manipulation for novel objects when instructed in natural language ('push the can to the left'), achieving 68% success on unseen objects compared to 45% for RT-1. The key insight is that internet-scale pretraining provides implicit physics understanding — the VLM has seen millions of images of objects being pushed and can reason about likely outcomes. However, precise quantitative push prediction still requires task-specific data, as foundation models trade precision for breadth.

Push data collection requires precise tracking of both the pusher (robot finger or tool) and the object before, during, and after each push interaction. The standard setup uses an overhead camera system (RGB or RGB-D) with fiducial markers (AprilTags) on the objects for sub-millimeter pose tracking at 30-60 Hz. The pusher position is recorded from robot joint encoders at 100+ Hz. For systematic push data collection, the workspace should be a flat, uniform surface with known friction properties — anodized aluminum or laminate surfaces provide consistent friction coefficients across the workspace.

Systematic push data collection varies push parameters across a grid: push location on the object boundary (8-16 evenly spaced points), push angle relative to the object surface normal (0, 15, 30, 45 degrees), and push velocity (10-100 mm/s in 3-5 steps). For each parameter combination, record the full push interaction: pre-push object pose, push start position, push trajectory (position over time), and post-push object pose after settling. This systematic approach produces 100-500 pushes per object in 1-2 hours, with complete coverage of the push parameter space.

For goal-directed push manipulation demonstrations, teleoperation is preferred over systematic collection because it captures human push planning strategies. Operators are presented with a current object configuration and a target configuration (shown as an overlay or separate display) and must push objects to achieve the goal. Record the full push sequence (typically 3-15 pushes for multi-object arrangements) with per-push annotations: push start/end positions, contact duration, object pose change, and distance to goal after each push. Include both successful and unsuccessful demonstrations — failed push sequences where the operator overshoots or creates unrecoverable configurations provide valuable negative signal.

For multi-object push manipulation (singulation, sorting, arrangement), the workspace should contain 5-15 objects in randomized configurations. Operators push objects to achieve specified goals: separate touching objects (singulation), sort objects by category into designated zones, or arrange objects into a target pattern. Annotate per-push: which object was pushed, whether any other objects were displaced by the push (cascading effects), and the post-push state of all objects. Multi-object push data is particularly valuable because it teaches policies about object-object interactions during pushing — a pushed object can collide with and displace neighboring objects, creating complex chain reactions.

Claru provides pushing data collection for both systematic push physics research and goal-directed manipulation applications. Our push collection stations feature overhead RGB-D cameras with AprilTag fiducial tracking at sub-millimeter accuracy, calibrated planar surfaces with characterized friction coefficients, and robot arms instrumented with fingertip force sensors for measuring push contact forces. We support both systematic parameter-sweep collection (100-500 pushes per object per hour) and teleoperated goal-directed push demonstrations.

We collect pushing data on client-supplied objects across diverse shapes, materials, and weight distributions. For systematic datasets, we vary push location, angle, and velocity across a configurable parameter grid with automated collection protocols. For goal-directed push manipulation, our operators demonstrate multi-step push sequences for singulation, sorting, and arrangement tasks with per-push annotations including contact points, displacement vectors, and goal-distance metrics.

Claru delivers pushing datasets formatted for forward push prediction models, goal-conditioned RL environments, Diffusion Policy training, or custom architectures. Standard deliverables include pre/post-push object poses with sub-millimeter accuracy, push parameter annotations (location, angle, velocity, duration), contact force profiles, and for multi-step demonstrations, per-push state annotations and goal achievement labels. Our collection throughput enables rapid scaling from research-scale datasets (10K-50K pushes) to production-scale corpora (200K+ interactions).