How to Annotate Hand-Object Interactions in Video

Before touching any video, define exactly what you are annotating and at what granularity. Hand-object interaction annotation spans multiple layers, and trying to annotate everything simultaneously burns time and creates inconsistent data. Choose from these annotation layers based on your downstream task:

(1) Hand pose: 21 keypoints per hand following the MediaPipe/MANO convention (wrist + 4 joints per finger). Each keypoint has (x, y, z) in camera coordinates plus a visibility flag (visible, self-occluded, object-occluded, out-of-frame). (2) Grasp type: per-frame or per-clip label from a defined taxonomy. The Feix taxonomy defines 16 grasp types (precision pinch, power wrap, lateral tripod, etc.) and is the most widely adopted in robotics. For kitchen/household tasks, a simplified 6-type taxonomy (pinch, wrap, hook, press, twist, no-contact) often suffices. (3) Contact annotation: binary mask or sparse keypoints indicating which parts of the hand are in physical contact with the object surface. At minimum, annotate contact per finger (thumb, index, middle, ring, pinky) as a boolean per frame. (4) Temporal action boundaries: start/end timestamps for manipulation primitives (approach, contact, manipulate, release, retreat).

Create a formal annotation specification document with visual examples for each label. For grasp types, include reference photos showing the canonical hand configuration. For temporal boundaries, define the exact frame where each phase begins (e.g., "contact starts at the first frame where any finger touches the object surface, visible as deformation or cessation of relative motion"). Ambiguous definitions are the primary source of inter-annotator disagreement.

Use a state-of-the-art hand pose estimator to generate initial annotations that human annotators will then correct. This hybrid approach is 3-5x faster than fully manual annotation. The current best options are HaMeR (CVPR 2024, reconstructs MANO mesh from single images), FrankMocap (reconstructs both hand and body), and MediaPipe Hands (fastest, but 2D-only without a depth lifting step).

For HaMeR, install via `pip install hamer` and run batch inference: `python -m hamer.demo --img_folder /path/to/frames --out_folder /path/to/results --batch_size 16`. HaMeR outputs MANO parameters (pose, shape, translation) and 21 3D keypoints per detected hand. It also outputs a per-hand confidence score. On an NVIDIA A100, HaMeR processes approximately 30 fps at 640x480 resolution. For egocentric video where hands are large and central, accuracy is typically excellent (MPJPE under 10mm). For third-person video with small hands, accuracy degrades and you will need more human correction.

For each frame, extract and store: (a) 21 keypoints as a (21, 3) numpy array in camera coordinates, (b) hand bounding box as (x, y, w, h), (c) handedness (left/right), (d) overall confidence score (0-1), and (e) per-keypoint confidence. Save results as a JSON file per video clip with frame-level entries. Also generate a visualization overlay (skeleton drawn on the RGB frame) for each video so annotators can quickly scrub through and spot errors without needing to run inference themselves.

Compute quality statistics across your dataset: what percentage of frames have confident detections (score > 0.8), how many frames have no detection at all (hand fully occluded or out of frame), and the distribution of per-keypoint confidence. Frames with detection confidence below 0.5 should be prioritized for manual annotation.

Configure your annotation tool to display pre-computed hand poses overlaid on video and let annotators correct keypoints, add grasp labels, and mark temporal boundaries in a single pass. CVAT (Computer Vision Annotation Tool) is the best open-source option for this workflow because it supports video-native annotation with interpolation, skeleton templates, and attribute labels per frame.

In CVAT, create a project with a skeleton task type. Define a skeleton template matching the 21-keypoint MANO hand model: wrist as root, then chains of 4 keypoints for each finger (MCP, PIP, DIP, tip). Enable the 'outside' attribute for occluded keypoints. Import the pre-computed poses from Step 2 using CVAT's annotation import (convert your JSON to CVAT XML format using the `datumaro` library: `datum convert -if custom_json -of cvat -i preds/ -o cvat_import/`). This pre-populates the skeleton on each frame so annotators correct rather than draw from scratch.

Add attribute fields to the skeleton object for per-frame annotation: `grasp_type` (dropdown with your taxonomy values), `contact_fingers` (multi-select: thumb, index, middle, ring, pinky), and `manipulation_phase` (dropdown: approach, contact, manipulate, release, retreat, idle). Configure the timeline view so annotators can see the manipulation phase as a color-coded track below the video, making temporal boundary annotation feel like editing a video timeline rather than labeling individual frames.

Set up the annotation workflow with three stages: (1) Pose correction: annotator adjusts keypoint positions and marks occlusion flags, spending approximately 2-5 seconds per frame on frames with pre-annotation and 10-15 seconds on frames without. (2) Grasp and contact labeling: annotator scrubs through the corrected video and adds grasp type + contact finger labels at key transitions, using interpolation for stable grasps. (3) Review: a second annotator verifies a random 20% sample. Configure CVAT's project to enforce this ordering using task stages.

Beyond hand pose, annotate the interaction semantics that make the data useful for robot learning. This step adds three layers: fingertip contact points on the object surface, object identity and state, and manipulation intent.

For contact annotation, the most practical approach for video data is per-finger binary contact labels per frame. Create a timeline annotation track for each finger (thumb through pinky). An annotator marks the exact frame where each finger makes and breaks contact with the object. For higher-fidelity contact, annotate contact regions on the object surface using 2D bounding circles (center point + radius in pixels) around each contact area. This is especially important for dexterous manipulation tasks where the exact contact location on the object determines the manipulation outcome (e.g., grasping a mug by the handle vs. the body).

For object annotation, use a lightweight tracking approach. Draw a bounding box around the manipulated object on the first frame, assign it a category label (e.g., `mug`, `bottle`, `knife`) and an instance ID, then use a semi-automatic tracker (CSRT or SiamRPN in OpenCV) to propagate the box through the clip. Annotators verify and correct the tracked box every 10-20 frames. Include object state attributes where relevant: `empty/full` for containers, `open/closed` for doors and drawers, `on/off` for switches.

For manipulation intent, annotate a natural language description of the action per temporal segment. Keep descriptions atomic and structured: "[hand] [verb] [object] [modifier]", e.g., "right hand grasps mug by handle", "left hand presses lid down", "right hand rotates jar counterclockwise". These language annotations are directly consumable by vision-language-action models like RT-2 and OpenVLA that condition on natural language task descriptions. Aim for 5-15 words per segment. Store the language annotation as a separate track aligned to the temporal boundaries from Step 3.

This is the most time-intensive annotation layer. Budget 3-5 minutes per 10-second clip for contact + object + language annotation on top of the corrected hand pose.

Run a multi-stage quality assurance pipeline before the annotated data enters your training set. For hand pose annotations, implement four automated checks:

(1) Bone-length consistency: compute the distance between consecutive joints for each finger across all frames. Bone lengths should be constant (within 5% of the per-clip median) since the hand skeleton does not change size. Frames where any bone length deviates by more than 10% indicate a keypoint was dragged to the wrong location. Flag these automatically: `bone_lengths = np.linalg.norm(kp[1:] - kp[:-1], axis=-1)` then check `np.abs(bone_lengths - np.median(bone_lengths, axis=0)) > 0.1 * np.median(bone_lengths, axis=0)`.

(2) Self-intersection: check that no two non-adjacent finger segments cross. Compute the minimum distance between each pair of finger bone segments using line-segment distance formulas. Distances under 5mm between non-adjacent fingers indicate an anatomically impossible configuration.

(3) Temporal smoothness: compute the per-keypoint velocity between consecutive frames. Velocities exceeding 500mm per frame (at 30fps) indicate a keypoint jump, likely from a tracker failure or annotation error. Use a Savitzky-Golay filter (`scipy.signal.savgol_filter` with window=5, polyorder=2) to smooth keypoint trajectories, then flag frames where the raw-to-smoothed distance exceeds 10mm.

(4) Contact consistency: when a finger is labeled as in contact with the object, verify that the fingertip keypoint is within 15mm of the object bounding box. A contact label with the finger clearly in the air is either a contact annotation error or a pose annotation error.

For inter-annotator agreement on grasp types, compute Cohen's kappa on the 20% overlap subset. Target kappa > 0.75 for the full taxonomy or > 0.85 for a simplified taxonomy. For temporal boundaries, compute the mean offset between annotators' boundary frames and target under 3 frames (100ms at 30fps). Generate a per-annotator quality dashboard and retrain underperforming annotators on the specific error patterns they exhibit.

Package the validated annotations into formats that integrate directly with robot learning pipelines. The output for each video clip should be a single structured record containing synchronized streams: RGB frames, 21-keypoint hand pose per frame (left and right), grasp type labels, contact flags, temporal boundaries, object tracks, and language descriptions.

For RLDS format (consumed by RT-X, Octo, OpenVLA), structure each clip as an episode in a TFRecord. Store hand keypoints as `observation/hand_keypoints_left` (float32, shape [21, 3]) and `observation/hand_keypoints_right` (same). Store grasp type as `observation/grasp_type` (int32 index into your taxonomy). Store contact as `observation/contact_fingers` (bool, shape [5] for thumb through pinky). Store language as `language_instruction` (string). Use `tfds.features.FeaturesDict` for the episode spec and `tensorflow_datasets.builder` to create the dataset.

For HDF5 format (consumed by Diffusion Policy, ACT, MimicGen), organize each clip as a group: `/clip_N/rgb` (uint8, T x H x W x 3), `/clip_N/hand_pose_left` (float32, T x 21 x 3), `/clip_N/hand_pose_right` (float32, T x 21 x 3), `/clip_N/grasp_type` (int32, T), `/clip_N/contact` (bool, T x 2 x 5 for left/right x fingers), `/clip_N/language` (string attribute), `/clip_N/temporal_segments` (structured array with start_frame, end_frame, phase_label). Use chunked compression: `chunks=(min(T, 100), 21, 3), compression='lzf'` for keypoints.

Provide a Python dataloader class with these methods: `__len__()` returning the number of clips, `__getitem__(idx)` returning a dictionary of tensors for one clip, and `get_statistics()` returning per-feature means and standard deviations for normalization. Include a visualization script that takes a clip index and renders an MP4 with skeleton overlay, contact indicators (green dots on contacting fingers), and the language description as a text overlay. This visualization is the single most important deliverable for building trust with downstream users.

Generate a dataset card (Markdown file) documenting: total clips, total frames, camera specs, annotation layers, taxonomy definitions, inter-annotator agreement scores, known limitations (e.g., 'left-hand annotations are lower quality due to occlusion bias in egocentric video'), and citation information.