Keypoint Annotation: Landmark Labeling for Pose Estimation and Robot Learning

Keypoint annotation is the process of marking specific, semantically defined landmark points on objects, human bodies, or hands in images and video frames. Each keypoint is a 2D coordinate (x, y) paired with a visibility flag indicating whether the point is visible, occluded but inferrable from context, or absent from the frame. A set of keypoints connected by predefined edges forms a skeleton — a sparse but structured representation of an entity's spatial configuration.

The COCO keypoint format, which defines 17 body landmarks (nose, left/right eye, ear, shoulder, elbow, wrist, hip, knee, ankle), has become the de facto standard for human pose datasets. Extensions like HALPE (136 keypoints including hand and foot joints) and Whole-Body COCO (133 keypoints) provide finer granularity for applications that require hand and facial landmark detail. For hands specifically, the 21-joint model (wrist plus 4 joints per finger) used by MediaPipe and FreiHAND has become standard.

In robotics, keypoint annotations serve multiple functions beyond pose estimation. Object keypoints enable category-level 6-DOF pose estimation, where a model predicts the 3D position and orientation of any object within a category from a single RGB image — KeypointDeformer (Jakab et al., 2021) and NOCS (Wang et al., 2019) pioneered this approach. Keypoints also define grasp affordances: specific points on an object that indicate where and how to grasp it. For articulated objects like doors or drawers, keypoints mark joint axes and handle locations that determine the manipulation strategy.

The quality of keypoint annotations is measured by mean per-joint position error (MPJPE) against expert ground truth, typically reported in pixels for 2D annotations or millimeters for 3D. State-of-the-art pose estimation models achieve MPJPE of 3-5 pixels on standard benchmarks, which sets the target accuracy for training annotations — annotations less accurate than the model's target performance provide contradictory training signal.

Designing a keypoint annotation protocol for robotics requires three key decisions: keypoint schema definition, visibility handling, and quality assurance methodology.

Keypoint schema definition determines which points are annotated and how they connect. For standard body and hand pose, established schemas (COCO-17, HALPE-136, MediaPipe-21) should be used directly to enable transfer learning from large pre-existing datasets. For object keypoints, custom schemas must be designed per object category, selecting points that are visually identifiable from multiple viewpoints, semantically meaningful for the manipulation task, and consistently locatable across object instances within the category. A good practice is to define 8-12 keypoints per object category, including at least 4 non-coplanar points to enable unique 6-DOF pose recovery through PnP algorithms.

Visibility handling is critical for real-world robotics data where occlusion is the norm rather than the exception. The standard three-level visibility flag (visible / occluded-but-labeled / not-in-frame) must be consistently applied. For occluded keypoints, annotators estimate the position based on body knowledge (for human joints), geometric reasoning (for object points), or temporal context (for video sequences where the point was recently visible). Training annotators to accurately place occluded keypoints is the single highest-impact quality investment — models trained with accurate occluded keypoint labels significantly outperform those trained only on visible keypoints.

Quality assurance for keypoint annotation combines automated checks with human review. Automated checks include anatomical plausibility validation (bone segment lengths within physiological ranges, joint angles within biomechanical limits), temporal consistency checks for video (keypoint displacement between frames below physically plausible velocity thresholds), and left-right symmetry checks. Human review targets the hardest cases: heavily occluded joints, ambiguous hand configurations, and edge cases where annotators frequently disagree. Claru computes per-keypoint agreement metrics across multiple annotators, identifying specific joints that require guideline refinement or additional annotator training.

Claru delivers production-grade keypoint annotation across the three domains central to physical AI: body pose, hand pose, and object landmarks. Our annotation pipeline uses model-assisted pre-labeling — ViTPose for body skeletons, MediaPipe for hand joints — with human correction that focuses annotator effort on the occluded and ambiguous joints where models are weakest. For object keypoints, Claru collaborates with each client to define custom keypoint schemas aligned to their manipulation tasks, ensuring the annotated landmarks carry direct geometric meaning for grasp planning and pose estimation. Quality controls include anatomical plausibility checks, temporal consistency validation for video sequences, and multi-annotator agreement monitoring with a target MPJPE below 3 pixels. Our 10,000+ trained annotators across 100+ cities produce keypoint data at scale across diverse environments and object categories, delivering the volume and variety that production pose estimation systems require.