Inter-Annotator Agreement: Measuring and Ensuring Training Data Quality

Inter-annotator agreement (IAA) is a family of statistical measures that quantify the degree to which independent human annotators assign the same labels to the same data. It is the foundational quality metric for any supervised learning dataset, because a model trained on human labels can only be as reliable as those labels are consistent. If annotators cannot agree on the correct label for an example, the model receives contradictory training signal and learns an incoherent decision boundary.

The simplest IAA measure is percent agreement — the fraction of items on which all annotators assigned the same label. However, percent agreement is misleading because it does not account for chance: two annotators randomly labeling binary data will agree 50% of the time by chance alone. Cohen's kappa (1960) addresses this by computing the ratio of observed agreement above chance to maximum possible agreement above chance, yielding a value from -1 (perfect disagreement) to 1 (perfect agreement), with 0 indicating chance-level agreement.

For production annotation pipelines involving dozens or hundreds of annotators who each label different subsets of the data, Krippendorff's alpha (1970, 2004) is the preferred metric. It generalizes across any number of annotators, handles missing data (not every annotator sees every item), and supports nominal, ordinal, interval, and ratio measurement scales. For spatial annotations like bounding boxes and segmentation masks, agreement is measured geometrically: intersection over union (IoU) between annotator masks, or normalized point distance for keypoints.

IAA serves three critical functions in data engineering. First, it validates annotation guidelines: if trained annotators cannot agree, the guidelines are ambiguous and must be revised. Second, it identifies individual annotators who need retraining or removal. Third, it sets an empirical ceiling on model performance — a model evaluated on labels from a process with 0.80 kappa cannot reliably exceed 0.80 accuracy on the same label distribution, because the evaluation labels themselves contain 20% noise.

Implementing IAA measurement in a production annotation pipeline requires deliberate design decisions at three levels: overlap strategy, metric selection, and intervention protocols.

Overlap strategy determines which items receive multiple annotations. Full overlap (every annotator labels every item) provides the most complete IAA measurement but is cost-prohibitive at scale. The standard practice is partial overlap: 10-20% of items are assigned to multiple annotators, selected either randomly or stratified by expected difficulty. For a 100,000-item dataset, a 15% overlap means 15,000 items receive duplicate annotations — enough to compute statistically significant agreement scores per annotator, per category, and per difficulty stratum. Items with high annotator disagreement are flagged for expert adjudication, where a senior annotator provides the final label.

Metric selection depends on the annotation type. For binary and multi-class classification, Cohen's kappa (two annotators) or Fleiss' kappa (three or more annotators) is appropriate. For ordinal scales (quality ratings, severity levels), weighted kappa or Krippendorff's alpha with ordinal distance function captures the magnitude of disagreement — rating an item 3 versus 4 is a smaller error than rating it 1 versus 5. For spatial annotations, IoU is the standard geometric agreement metric, computed per instance. For keypoint annotations, mean per-joint position error (MPJPE) in pixels or millimeters measures spatial consistency. The choice of metric must be locked before annotation begins and documented in the dataset card.

Intervention protocols define what happens when IAA drops below thresholds. Claru implements a three-tier system: (1) annotator-level alerts trigger when an individual's rolling agreement falls below the project threshold, prompting targeted retraining on examples they misjudged; (2) category-level alerts trigger when a specific label class shows low agreement, indicating ambiguous guidelines that need revision with additional examples and edge case clarifications; (3) dataset-level alerts trigger when overall agreement trends downward, signaling guideline drift or annotator fatigue. Each tier has a response protocol that is executed within 24 hours to prevent contamination of the broader dataset.

Inter-annotator agreement is the backbone of Claru's quality assurance system. Every Claru annotation project measures IAA continuously rather than as a one-time validation, using Krippendorff's alpha for categorical labels and IoU for spatial annotations. Our annotator qualification pipeline requires 85%+ agreement with gold-standard labels before any annotator enters production. During active annotation, 10-15% random overlap generates continuous IAA signals, and annotators whose rolling scores fall below project thresholds receive immediate retraining. For physical AI datasets — where annotation types span keypoint labeling, action segmentation, grasp quality rating, and trajectory annotation — Claru computes IAA per annotation type and per object category, ensuring quality is uniform across the dataset. This rigor consistently delivers datasets with kappa above 0.80 for classification tasks and IoU above 0.85 for spatial annotations, giving our clients confidence that model performance is limited by architecture choices, not data quality.