How to Build an Object Tracking Dataset

Before any annotation begins, precisely define what gets tracked, how it gets labeled, and what metadata each track carries. Create an object category taxonomy that lists every trackable class relevant to your domain. For a kitchen robotics application, this might include: mug, plate, bowl, knife, spatula, cutting_board, food_item (with sub-categories for fruits, vegetables, proteins), hand_left, hand_right, and appliance. For warehouse robotics: box, pallet, forklift, worker, shelf, conveyor_item. Keep the taxonomy to 10-30 categories to maintain annotator accuracy. Each category needs a clear visual definition with 5-10 example images showing borderline cases.

Define the annotation format: axis-aligned bounding boxes (x, y, width, height) in MOT Challenge format, or rotated bounding boxes (x, y, width, height, angle) for elongated objects that benefit from tight fits, or segmentation masks in COCO format with RLE encoding. Specify the track metadata schema: each track has a unique integer ID (persistent across all frames where the object is visible), a category label, per-frame visibility flag (visible, partially occluded, fully occluded, out of frame), and per-frame bounding box or mask coordinates. Define rules for track lifecycle: a new track ID is created when an object first appears in the frame (not when it first exists in the scene). A track is terminated when the object permanently leaves the camera view. If the same object exits and re-enters the frame, it keeps the same ID if the gap is under 30 frames, but receives a new ID if the gap is longer (this matches standard MOT evaluation conventions).

Process your raw video into annotation-ready sequences and generate machine-assisted pre-annotations to accelerate human labeling. Extract videos at the target annotation frame rate using FFmpeg: ffmpeg -i input.mp4 -vf fps=15 -q:v 2 frames/%06d.jpg. Organize sequences into a directory structure that matches your annotation tool's expected format. For CVAT, create a task per video sequence with frames uploaded in order.

Generate pre-annotations using a two-stage pipeline. First, run an object detector (YOLOv8x or RT-DETR-L fine-tuned on your category taxonomy) on every frame to produce per-frame detections with class labels and confidence scores. Second, run a tracker (ByteTrack with default parameters: track_thresh=0.5, match_thresh=0.8, track_buffer=30) on the detections to assign preliminary track IDs. Export these pre-annotations in the annotation tool's import format (CVAT XML, Label Studio JSON, or MOT Challenge CSV). Pre-annotations reduce manual annotation time by 50-70% because annotators correct existing boxes rather than drawing from scratch.

Critical pitfall: pre-annotation quality directly impacts annotator behavior. If pre-annotations are mostly correct, annotators tend to accept them without careful inspection, missing subtle errors. If pre-annotations are frequently wrong, annotators lose trust and start ignoring them, negating the speed benefit. Calibrate your detector and tracker on a small test set before generating pre-annotations for the full dataset. The sweet spot is 70-85% precision: enough correct boxes to save time, but enough errors to keep annotators engaged. Include deliberately empty frames (where the detector missed obvious objects) as attention checks to verify annotators are not blindly accepting pre-annotations.

Structure annotation as a three-pass process: spatial annotation, temporal linking, and quality review. In the first pass (spatial), annotators draw or adjust bounding boxes on keyframes spaced 5-10 frames apart, ensuring tight fits around each object. In CVAT, use the Track mode which automatically enables linear interpolation between keyframes. Annotators should adjust the box on any frame where the interpolated position deviates from the true object position by more than 10% of the object size. This pass produces per-frame boxes with approximate track IDs.

In the second pass (temporal linking), a specialized annotator reviews each track's temporal consistency. They verify that the same physical object maintains the same track ID across the full sequence, correct any ID switches at occlusion boundaries, and fill in visibility flags for occluded frames. This pass is best performed by watching the video at 2-4x speed with track IDs overlaid on the visualization. Assign this pass to your most experienced annotators because temporal consistency errors (ID switches) are harder to detect and more damaging to tracker training than spatial inaccuracy.

In the third pass (quality review), a senior annotator or QA specialist reviews a stratified 15-20% sample of the annotated sequences. They check for: boxes that are consistently too loose or too tight (more than 15% padding or cutting into the object), ID switches at occlusion events, missing tracks (objects present in the frame but not annotated), ghost tracks (annotations on locations where no object exists), and incorrect category labels. Compute per-annotator error rates from this review and use them to weight annotator compensation and assign retraining. Target an error rate below 5% of boxes per frame after the three-pass process. Any sequence with an error rate above 10% should be fully re-annotated rather than patched.

Occlusion events are the highest-value annotations in a tracking dataset because they test the tracker's ability to maintain identity through visual disappearance. Develop detailed annotation guidelines for each occlusion scenario and verify compliance through targeted audits.

For partial occlusion (object still partially visible): annotate the visible portion of the object. If using bounding boxes, draw the box around the full estimated extent of the object including the occluded region, and set the visibility flag to partially_occluded. If using masks, annotate only the visible pixels. This distinction matters because some trackers use the estimated full bounding box for association while others use only the visible mask for appearance matching.

For full occlusion (object completely hidden behind another object): cease annotating the bounding box or mask, set the visibility flag to fully_occluded, and crucially maintain the track ID in the annotation metadata so that when the object reappears, it can be re-linked. Provide annotators with a playhead that shows upcoming frames so they can anticipate where the object will reappear and maintain the correct ID.

For out-of-frame events: when an object moves outside the camera's field of view, set the visibility flag to out_of_frame. If the object re-enters within 30 frames (1-2 seconds at 15-30 FPS), reuse the same track ID. For longer absences, create a new track ID. Document this threshold in the annotation spec because it directly affects IDF1 evaluation scores.

For object state changes: some objects change appearance during tracking (a box being opened, a mug being filled, food being cut). Maintain the same track ID through appearance changes as long as the physical object remains the same entity. Add an appearance_change event annotation at the frame where the change occurs. These annotations are particularly valuable for training re-identification features in tracking models.

Run a battery of automated consistency checks before finalizing the dataset. These checks catch errors that human review misses and provide quantitative quality scores for the entire dataset.

Temporal consistency check: for every track, compute IoU between the bounding box on frame t and frame t+1. Flag any instance where IoU drops below 0.5 between consecutive frames and the object is not flagged as occluded. This catches box jumps, ID switches, and annotation gaps. For a 10,000-frame dataset with 50 tracks, expect 2-5% of frame transitions to be flagged; review each flag and correct genuine errors.

Size consistency check: compute the bounding box area for each track over time and flag sudden size changes exceeding 50% between consecutive frames (excluding occlusion transitions). Objects in real video change size gradually due to camera motion and perspective, so sudden jumps indicate the box snapped to a different object or the annotator made an error.

Category consistency check: for each track, verify that the category label does not change across frames (it should not, since a mug does not become a plate). Any track with inconsistent categories has an ID switch embedded in it.

Coverage check: count the number of distinct objects visible in each frame (using a reference detector or manual count on a sample) and compare against the number of annotated tracks. Coverage should exceed 95%. Frames with significantly lower coverage contain missed annotations.

Compute aggregate quality metrics for the full dataset: mean boxes per frame, mean track length, occlusion event frequency, ID consistency score (fraction of tracks that pass the temporal consistency check), and category distribution. Publish these metrics in the dataset card. They serve as the contract between annotation and model training teams.

Export the validated annotations in standard tracking formats and generate evaluation-ready splits. The MOT Challenge format is the most widely adopted: a single CSV file per sequence with columns (frame_id, track_id, x, y, w, h, confidence, class_id, visibility). This format is directly compatible with py-motmetrics for MOTA/IDF1/HOTA evaluation and with most tracking libraries (mmtracking, TrackEval, norfair). For COCO-style evaluation, export annotations as a COCO JSON with the video_annotations extension that includes track_id per annotation.

Generate train/validation/test splits by held-out video sequences, not by sampling frames from the same videos. Hold out 15-20% of sequences for testing, ensuring the test set covers the full range of difficulty: include at least 2 easy sequences (few objects, no occlusion), 3 medium sequences (moderate object count, some occlusion), and 2 hard sequences (dense objects, frequent occlusion, fast motion). Create a splits.json manifest documenting the assignment.

Package the dataset with evaluation infrastructure. Include a Python evaluation script that loads predictions in MOT format, computes MOTA, IDF1, HOTA, MT (mostly tracked), ML (mostly lost), and FP/FN/ID-switch counts per sequence and averaged across the dataset. Include a visualization script that renders tracking results as color-coded bounding boxes overlaid on video frames, with track IDs displayed, exported as MP4 for easy review. Include a baseline: run ByteTrack with a YOLO detector on the test set and report the numbers, so downstream users have a concrete performance reference point. The dataset card should document: total frames, total tracks, frames per second, object categories, annotation type (box vs. mask), occlusion statistics, and the evaluation protocol (which metrics are primary, how confidence thresholds are handled).