COCO Format (Common Objects in Context): Complete Guide for Robotics Data

COCO annotations are stored as a single JSON file containing three core sections: images (an array of objects with id, file_name, width, and height), annotations (an array with id, image_id, category_id, bbox in [x, y, width, height] format, segmentation polygons or RLE-encoded masks, and area), and categories (an array with id, name, and supercategory). The bounding box convention uses top-left origin with width and height, distinguishing COCO from Pascal VOC's [xmin, ymin, xmax, ymax] format. The iscrowd flag differentiates individual instance annotations (polygon segmentation) from crowd regions (RLE mask).

The COCO format supports five distinct annotation types through separate JSON structures: object detection (bounding boxes), instance segmentation (per-pixel masks), keypoint detection (skeleton-based pose with visibility flags), panoptic segmentation (unified stuff and things), and image captioning (natural language descriptions). For keypoint detection, the categories array includes a keypoints field listing joint names and a skeleton field defining connectivity as pairs of keypoint indices. Each annotation carries a keypoints array of [x, y, visibility] triplets where visibility is 0 (not labeled), 1 (labeled but not visible), or 2 (labeled and visible). The panoptic extension, introduced by Kirillov et al. in 2019, uses PNG images where each pixel encodes a segment ID via R + G*256 + B*256*256, enabling both stuff (sky, road) and things (car, person) to be represented in a single annotation.

For robotics perception, the COCO format is particularly valuable because it standardizes the interface between annotation tools and model training frameworks. A single annotations JSON file can contain hundreds of thousands of annotations across thousands of images, with category hierarchies expressed through the supercategory field. The info section records dataset metadata (year, version, description, contributor, url, date_created), and the licenses array maps license IDs to names and URLs. This self-describing structure means any tool that reads COCO format can ingest any COCO-compatible dataset without format-specific adapters.

The pycocotools library (pip install pycocotools) is the standard API for working with COCO annotations. Initialize with coco = COCO('annotations.json'), then query annotations by image with coco.getAnnIds(imgIds=[img_id]) or by category with coco.getAnnIds(catIds=[cat_id]). The API handles both polygon and RLE segmentation formats transparently, and coco.annToMask(ann) converts any annotation to a binary mask. For evaluation, COCOeval computes the standard AP metrics at IoU thresholds from 0.5 to 0.95 in steps of 0.05, giving the COCO-standard mAP that is the de facto reporting metric in detection research.

Creating a new COCO annotation file follows a straightforward pattern: build the images, annotations, and categories arrays as Python lists of dictionaries, then serialize with json.dump(). Each annotation must have a unique id (typically a running integer), and the image_id must reference a valid entry in the images array. For segmentation, polygon points are stored as a flat list [x1, y1, x2, y2, ...] representing one or more closed polygons, and the area field should be computed from the segmentation mask using pycocotools.mask.area(). The bbox field must be [x, y, w, h] where (x, y) is the top-left corner. Tools like FiftyOne (fiftyone.zoo.datasets) and CVAT provide COCO export with correct formatting, while Label Studio and Labelbox offer one-click COCO JSON export.

For large-scale annotation projects, the COCO Results Format enables submitting model predictions separately from ground truth. A results file is a JSON array where each entry contains image_id, category_id, score, and the detection-specific fields (bbox for detection, segmentation for instance segmentation, keypoints for pose). This separation of predictions from ground truth allows evaluation without modifying the original annotation file and enables server-side evaluation on held-out test sets, which is how the COCO Challenge leaderboard operates.

In robotics applications, COCO format serves as the standard for perception annotations that feed into grasping, navigation, and manipulation pipelines. Object detection annotations in COCO format are used to train the perception modules of systems like GraspNet, Contact-GraspNet, and AnyGrasp, where accurate bounding boxes and segmentation masks are prerequisites for grasp pose estimation. For bin picking and warehouse automation, COCO instance segmentation provides the per-object masks needed to separate overlapping items before grasp planning.

Keypoint annotations in COCO format extend naturally to robotic manipulation of articulated objects. The skeleton definition allows representing hinge joints on cabinet doors, drawer handles, and tool grips as connected keypoint graphs. The SAPIEN and PartNet-Mobility benchmarks define keypoint schemas for articulated household objects following COCO conventions. For human-robot collaboration, COCO keypoint annotations trained on the COCO-WholeBody dataset (133 keypoints covering body, face, hands, and feet) provide the pose estimation foundation for safe proximity monitoring and handover planning.

The COCO evaluation metrics (AP@[.5:.95], [email protected], [email protected], AR@[1,10,100]) have become the universal reporting standard for detection and segmentation. In robotics, these metrics are supplemented with task-specific evaluations (grasp success rate, pick-and-place accuracy), but COCO AP remains the accepted baseline for comparing perception model quality. The pycocotools.cocoeval module computes these metrics efficiently, handling edge cases like crowd annotations, small/medium/large object splits, and per-category AP breakdowns.

Claru delivers object detection, instance segmentation, panoptic segmentation, and keypoint annotations in COCO format. Every annotation file is validated against the official COCO schema using pycocotools to ensure correct annotation IDs, bounding box consistency, segmentation mask integrity, and category mapping. Annotations are COCO-compatible for direct use with Detectron2, MMDetection, Ultralytics YOLO, and any framework that accepts COCO JSON.

For robotics perception projects, Claru provides custom category taxonomies aligned with your object inventory, whether that is warehouse SKUs, kitchen utensils, industrial parts, or articulated household objects. Keypoint annotations include application-specific skeleton definitions for grasp points, functional parts, and articulation joints. Large deliveries include per-split annotation files (train, val, test) with stratified category distributions and a statistics report documenting per-category instance counts, bbox size distributions, and annotation density per image.