How to Build a Contact-Rich Manipulation Dataset

Contact-rich manipulation spans a wide range of task complexities. Before collecting data, define which contact regimes your dataset needs to cover and configure sensors accordingly.

Categorize your target tasks by contact complexity. Level 1 -- quasi-static contact: tasks where contact forces change slowly and predictably (placing an object on a surface, sliding a tray). These require force/torque sensing at 100 Hz minimum. Level 2 -- dynamic contact: tasks with rapid force transitions (inserting a peg, closing a latch, screwing a lid). These require 500+ Hz force sensing and precise temporal alignment with vision. Level 3 -- compliant contact: tasks involving deformable objects or environments (wiping a surface, folding cloth, kneading dough). These require distributed force sensing or tactile sensors to capture the spatial contact pattern, not just the resultant wrench.

For the force/torque sensor, mount it between the robot's wrist flange and the gripper. Use a rigid coupling (not a compliant mount) to preserve high-frequency force signals. Calibrate the sensor to compensate for the gripper's weight: with the gripper attached but no object, record the wrench at 10 different orientations spanning SO(3), then compute the gravity compensation parameters. In ROS2, use the `force_torque_sensor_broadcaster` and publish compensated wrench on `/ft_sensor/wrench_compensated` at the sensor's native rate.

For tactile sensors (GelSight Mini or DIGIT), mount them on the gripper fingertips. GelSight Mini connects via USB and produces 320x240 RGB images of the elastomer deformation at 30 fps. Calibrate by pressing against a flat surface with known force (using the F/T sensor as reference) and recording the resulting tactile image. This creates a force-to-tactile-image mapping useful for downstream supervision.

Set up the camera configuration: one wrist-mounted RealSense D435 for close-range manipulation view (640x480 at 30fps) and one or two external cameras for workspace overview. Ensure the wrist camera does not occlude the tactile sensors and that the external cameras have a clear view of the contact zone. Verify all sensor streams are accessible in ROS2 with `ros2 topic list` and check publication rates with `ros2 topic hz`.

Contact-rich data has the tightest synchronization requirements of any manipulation dataset. A 30ms misalignment between force and vision -- invisible in most tasks -- can cause a contact-aware policy to associate the wrong force with the wrong visual state, learning incorrect contact dynamics.

Implement hardware-triggered synchronization. The gold standard is to use the F/T sensor's data-ready signal as the master clock. The ATI NetFT outputs a data-ready pulse on each new sample. Connect this to a hardware trigger that simultaneously triggers the camera capture. With the RealSense SDK, use `rs2::context::set_devices_changed_callback` and external trigger mode (via GPIO on a companion microcontroller). This guarantees force and vision timestamps align to within 1ms.

If hardware triggering is not feasible, use ROS2's software synchronization. Record all topics to a rosbag2 file with `ros2 bag record -a`. After recording, align data in post-processing: resample force/torque data (500+ Hz) to match camera timestamps (30 Hz) using linear interpolation. Compute the cross-correlation between force magnitude and visual features (like gripper aperture change) to estimate and correct any systematic clock offset between sensors.

Structure the recording pipeline as a ROS2 launch file that starts all sensor drivers, the recording node, and a real-time monitoring dashboard. The dashboard should display: camera feeds, force/torque magnitude bar graphs, gripper state, and a recording status indicator. Include automatic episode segmentation: the pipeline detects episode start (gripper opens from home position) and episode end (gripper returns to home position) and writes separate files per episode.

For each episode, record: RGB images (wrist + external cameras), depth images (wrist camera), 6-DoF force/torque at native sensor rate, joint positions and velocities at 100+ Hz, end-effector pose (position + quaternion) at 100+ Hz, gripper width, gripper force (if available), tactile images (if using GelSight/DIGIT), and operator's control inputs (for teleoperation data). Validate the recording by replaying the first episode and verifying all streams are present, at the expected rates, and temporally aligned.

Contact-rich tasks require teleoperators who can feel the contact forces, not just see them. Standard VR teleoperation (Oculus controllers) provides no haptic feedback, which leads to excessive forces and unrealistic contact behaviors. Use a teleoperation system with force feedback or at minimum, a real-time force display.

The best system for contact-rich collection is bilateral teleoperation: the operator controls a leader robot that mirrors the follower robot's motion and reflects its contact forces back to the operator's hand. The GELLO system (Stanford, 2024) provides a low-cost bilateral interface using a 3D-printed replica of the follower arm with torque-controlled joints. The operator physically moves the GELLO arm, and force feedback from the follower's F/T sensor is rendered as resistance in the GELLO joints. This gives the operator direct force sensation.

If bilateral teleoperation is not available, use VR teleoperation with a real-time force overlay. Display the F/T sensor readings as a 3D force vector rendered in the VR scene at the gripper location. Color-code by magnitude: green (< 5N), yellow (5-15N), red (> 15N). Train operators to keep forces in the green/yellow range and to recognize when they are applying excessive force. This is less natural than haptic feedback but still produces better contact data than blind teleoperation.

Design the collection protocol for contact-rich tasks with specific attention to force profiles. For each task, define the expected force profile: peg insertion should show a low approach force (< 2N), a contact spike at the chamfer entry (5-10N), a sustained insertion force (3-8N), and a release to zero. During pilot collection, plot the actual force profiles and compare to expectations. If operators consistently produce forces 3x higher than expected, the teleoperation gain needs adjustment or the operators need more training.

Collect at least 20% negative examples per task: episodes where excessive force was applied, where the insertion failed, or where the object was damaged. These are critical for training force-aware policies that learn contact limits. Label negative examples during collection by having the operator press a button to flag the current episode.

Run collection in 30-minute sessions with 10-minute breaks. Contact-rich teleoperation causes operator fatigue faster than non-contact tasks because the operator must maintain precise force control throughout. Track per-operator metrics: task completion time, average contact force, maximum force, and success rate. Retire operators whose maximum forces consistently exceed the safety threshold.

Contact-rich data requires annotation layers beyond standard trajectory annotation. The key annotations are: contact event detection, contact phase segmentation, force quality scoring, and contact geometry labeling.

Contact event annotation marks the exact timestep of each contact transition: free-space-to-contact (approach), contact-to-free-space (release), and contact-type transitions (sliding-to-sticking, grasping-to-insertion). Implement semi-automatic detection: compute the derivative of force magnitude (`df = np.abs(np.diff(force_magnitude))`), threshold at 0.5 N/timestep to detect contact onsets, and present detected events to human annotators for verification and classification. The annotator confirms each detected event and assigns a contact type: {initial_contact, stable_grasp, sliding, insertion, pressing, release}.

Force phase segmentation divides each episode into phases based on the dominant contact regime. For a peg insertion task: free_approach (no contact, F < 0.5N), alignment (light contact, F = 0.5-3N with lateral correction forces), insertion (sustained forward force, Fz = 3-10N with Fx/Fy near zero), seating (force spike as peg bottoms out, Fz > 10N briefly), and release (force returns to zero). Annotate phase boundaries on the force-time plot, which is faster and more precise than using video alone. The force signal makes boundaries unambiguous -- a human annotator can segment a 30-second insertion episode in under 60 seconds using the force plot.

Force quality scoring rates each episode's contact behavior on a 1-5 scale: (1) excessive force, likely to damage real objects; (2) high force but successful; (3) acceptable force, some unnecessary contact; (4) clean contact, minimal unnecessary force; (5) expert-level contact, smooth force profiles with no spikes. This quality score enables weighted training where high-quality episodes receive higher loss weight.

For tasks involving contact geometry (insertion, fitting, assembly), annotate the contact configuration: which surface of the gripper/finger contacts which surface of the object. For GelSight tactile data, annotate the tactile image with a contact region mask and estimate the contact normal direction. This geometric annotation is expensive (5-10 minutes per episode) and should be reserved for a subset of episodes used for contact-model training.

Contact data has unique quality failure modes that are not present in vision-only datasets. Implement automated checks specifically for force/torque and tactile data.

F/T sensor validation: check for sensor drift by comparing the zero-force baseline at episode start vs. episode end (with the gripper in free space). Drift exceeding 0.5N or 0.05Nm indicates thermal drift or a calibration issue. Re-zero the sensor between episodes by recording a 1-second free-space baseline and subtracting it. Check for sensor saturation: any timestep where the force or torque equals the sensor's maximum range exactly (e.g., 145.0N for the ATI Mini45) indicates clipping and the true force is unknown. Flag saturated episodes.

Synchronization validation: compute the cross-correlation between force magnitude and gripper aperture change. The correlation should peak at lag=0 (within one camera frame). If the peak is at lag=2 frames (66ms at 30fps), the force data is systematically delayed and needs time-shifting. Run this check on every recording session. Compute `lag = np.argmax(np.correlate(force_mag, np.abs(np.diff(gripper_width)), mode='full')) - len(force_mag)` and verify `abs(lag) <= 1`.

Contact physics validation: verify that forces obey basic physical laws. During free-space motion (no contact), the measured wrench should equal the gravitational load of the gripper plus payload (within noise). Forces exceeding 2N during annotated free-space phases indicate a synchronization error (the contact event was annotated too late) or a sensor calibration error. During stable grasps, the grip force should be constant within 10%. High-frequency force oscillations during a supposedly stable grasp indicate a loose grip, sensor noise, or vibration.

Tactile data validation: check that tactile images change when the F/T sensor reports contact and remain at baseline when no contact is detected. Compute the pixel-wise difference between each tactile image and the no-contact reference image. The summed difference should correlate with the F/T force magnitude (Pearson r > 0.6 for GelSight data). If tactile and force data are uncorrelated, the tactile sensor may be occluded, miscalibrated, or not in contact.

Generate a per-episode quality report with pass/fail on each check. Episodes that fail drift, saturation, or synchronization checks should be excluded from training data. Episodes that fail physics checks should be reviewed manually -- they may contain interesting edge cases (object slipping, unexpected collisions) that are valuable if correctly annotated.

Contact-rich datasets are among the most complex to package because they contain high-frequency multi-modal data that must remain synchronized after export. Design the storage format to preserve temporal alignment and enable efficient access to individual modalities.

Use HDF5 with per-episode groups and per-modality datasets. Structure: `/episode_N/obs/rgb_wrist` (uint8, T_cam x H x W x 3 at 30Hz), `/episode_N/obs/depth_wrist` (uint16, T_cam x H x W at 30Hz), `/episode_N/obs/force_torque` (float32, T_ft x 6 at 500Hz), `/episode_N/obs/tactile_left` (uint8, T_tac x 240 x 320 x 3 at 30Hz), `/episode_N/obs/joint_pos` (float64, T_proprio x 7 at 100Hz), `/episode_N/obs/ee_pose` (float64, T_proprio x 7 at 100Hz), `/episode_N/obs/gripper_width` (float64, T_proprio x 1 at 100Hz), `/episode_N/actions` (float64, T_action x A at the control frequency). Note the different T values per modality -- force/torque has 16x more timesteps than camera data.

Include a timestamps dataset for each modality: `/episode_N/timestamps/force_torque` (float64, T_ft), `/episode_N/timestamps/rgb_wrist` (float64, T_cam), etc. All timestamps are in the same clock domain (seconds since episode start). The dataloader uses these timestamps to align modalities on-the-fly: for a given camera frame at time t, the corresponding force/torque reading is `ft_data[np.argmin(np.abs(ft_timestamps - t))]`.

Store annotations in a separate group: `/episode_N/annotations/contact_events` (structured array: frame_index, contact_type, force_magnitude), `/episode_N/annotations/force_phases` (structured array: start_frame, end_frame, phase_label), `/episode_N/annotations/quality_score` (int, 1-5), `/episode_N/annotations/language_instruction` (string). Add episode-level metadata as attributes: `episode.attrs['task'] = 'peg_insertion'`, `episode.attrs['object'] = 'round_peg_8mm'`, `episode.attrs['operator_id'] = 'op_03'`, `episode.attrs['success'] = True`.

Provide a multi-modal dataloader that handles the different sampling rates. The key method: `get_observation(episode_idx, time_step, modalities=['rgb', 'force_torque', 'joint_pos'])` returns a dictionary with each modality interpolated or nearest-neighbor sampled to the requested timestep. For force/torque, provide both the instantaneous reading and a 100ms window around the timestep (as a 50-sample vector at 500Hz) to capture force dynamics that a single sample misses.

Include a force profile visualization script that plots all six F/T channels over time with contact event markers and phase color-coding. This is the primary debugging tool for anyone training on the data. Ship a Jupyter notebook that loads one episode, plots the force profile, overlays the contact annotations, and displays synchronized video frames at key contact events.