BOP Format (Benchmark for 6D Object Pose): Complete Guide for Robotics Data

BOP (Benchmark for 6D Object Pose Estimation) defines a standardized directory structure for storing RGB-D scenes with ground-truth 6D object poses. Each dataset is organized into scenes (numbered directories), where each scene contains scene_camera.json (per-frame camera intrinsics as 3x3 K matrix plus optional depth_scale), scene_gt.json (per-frame list of object annotations with obj_id, rotation matrix R as 9 floats in row-major order, and translation vector t in millimeters), and subdirectories for rgb/ (color images as PNG or JPEG), depth/ (16-bit PNG depth maps), and optionally mask/ and mask_visib/ (per-object binary masks distinguishing full object extent from visible portion). Object 3D models live in a separate models/ directory as PLY mesh files, indexed by models_info.json which stores per-model diameter, min/max XYZ extents, and symmetry annotations.

The BOP format encodes 6D pose as a rigid transformation from the model coordinate frame to the camera coordinate frame. The rotation matrix R is a 3x3 orthonormal matrix stored as 9 float values in row-major order, and the translation vector t is [tx, ty, tz] in millimeters from the camera optical center to the object origin. Camera intrinsics are stored as fx, fy (focal lengths in pixels), cx, cy (principal point in pixels), and depth_scale (multiplier to convert 16-bit depth values to millimeters, typically 0.1 or 1.0). This convention means that a 3D model point p_m in model coordinates transforms to camera coordinates as p_c = R * p_m + t, and projects to pixel coordinates as u = fx * p_c.x / p_c.z + cx, v = fy * p_c.y / p_c.z + cy.

The BOP Toolkit (bop_toolkit_lib) provides Python utilities for loading, visualizing, and evaluating BOP-format datasets. The standard evaluation metrics are VSD (Visible Surface Discrepancy), MSSD (Maximum Symmetry-Aware Surface Distance), and MSPD (Maximum Symmetry-Aware Projection Distance), computed at error thresholds that account for object symmetry. The BOP Challenge, running annually since 2017 at ECCV and ICCV workshops, has established these metrics as the community standard for pose estimation evaluation. As of BOP Challenge 2024, the benchmark includes 11 core datasets covering household objects (T-LESS, YCB-V, LM-O), industrial parts (ITODD), and textured objects (TUD-L, HB), with over 350,000 annotated object instances across 50,000+ images.

The bop_toolkit_lib Python package provides the standard API for BOP data. Loading a scene requires reading the JSON metadata files: scene_camera = json.load(open('scene_camera.json')), scene_gt = json.load(open('scene_gt.json')), where keys are string frame indices ('0', '1', ...). For each frame, scene_gt[str(frame_id)] returns a list of object annotations, each containing 'obj_id' (integer), 'cam_R_m2c' (9-element rotation matrix), and 'cam_t_m2c' (3-element translation in mm). The bop_toolkit_lib.inout module provides load_scene_gt(), load_scene_camera(), and load_ply() functions that handle format variations across BOP dataset versions.

Rendering BOP objects for training data augmentation is a critical workflow. The BOP Toolkit includes a rendering pipeline using Blender (via BlenderProc) or OpenGL that takes the 3D model PLY files and scene_gt.json poses to generate synthetic RGB-D images with pixel-perfect ground truth. BlenderProc, the recommended renderer for BOP, can generate photorealistic training data by compositing BOP object models onto random backgrounds with physically-based lighting. The BOP Challenge 2020 established that models trained on BlenderProc-generated synthetic data (PBR training images) can match or exceed models trained on real data alone, which has made synthetic-to-real transfer via BOP format a standard practice in pose estimation research.

For evaluation, the bop_toolkit_lib.score module computes VSD, MSSD, and MSPD metrics. VSD measures the visible surface discrepancy between the rendered ground-truth pose and the predicted pose, accounting for occlusion. MSSD and MSPD are symmetry-aware metrics that consider all equivalent poses for symmetric objects (e.g., a cylinder can be rotated around its axis). The BOP evaluation server at bop.felk.cvut.cz accepts result CSV files with columns scene_id, im_id, obj_id, score, R (rotation as 9 space-separated floats), t (translation as 3 space-separated floats), and time (inference time in seconds), providing standardized leaderboard evaluation.

While BOP was designed for pose estimation benchmarking, its precise 6D pose representation makes it directly applicable to robotic grasping and manipulation planning. Accurate 6D object pose is a prerequisite for grasp planning systems like GraspNet-1Billion, Contact-GraspNet, and AnyGrasp, which require knowing the exact position and orientation of target objects to compute stable grasp configurations. The BOP format's explicit camera intrinsics and depth maps enable seamless integration with depth-based grasp planners that operate in 3D camera coordinates.

For bin picking and warehouse automation, BOP-format data captures the exact scenario that pose estimation models must handle: multiple objects at known poses with varying levels of occlusion. The mask_visib/ directory explicitly encodes which portions of each object are visible, which is critical for training models that must handle heavy occlusion (objects stacked in bins). The T-LESS and ITODD BOP datasets specifically target industrial scenarios with texture-less objects and cluttered arrangements, making them directly relevant for factory automation perception pipelines.

The BOP format also supports the synthetic-to-real transfer workflow that has become standard in industrial robotics. Teams generate large-scale synthetic training data using BlenderProc with their specific object CAD models, outputting directly in BOP format. They then collect a smaller real-world validation set (also in BOP format) to evaluate sim-to-real transfer quality. The consistent format across synthetic and real data ensures that the same training and evaluation code works for both, reducing the engineering overhead of domain adaptation experiments.

Claru delivers 6D pose estimation data in BOP format with ground-truth poses computed from multi-view photogrammetry pipelines. For real-world data collection, we use multi-camera rigs with ArUco or AprilTag fiducial markers for initial pose estimation, followed by ICP (Iterative Closest Point) refinement against high-resolution 3D scans to achieve sub-millimeter translational and sub-degree rotational accuracy. Object 3D models are provided as textured PLY meshes captured via structured-light scanning or photogrammetry, with models_info.json populated with accurate diameter, extent, and symmetry metadata.

Every BOP delivery includes RGB images, registered 16-bit depth maps, per-object instance masks (both full and visible-only), and complete camera calibration. For teams requiring synthetic training data augmentation, we provide BlenderProc-compatible scene configurations that reproduce the lighting and camera conditions of the real collection environment. Large-scale deliveries follow the BOP Challenge sharding convention with separate scene directories, and all data is validated against the BOP Toolkit's integrity checker to ensure correct JSON formatting, consistent frame indexing, and valid rotation matrices (orthonormality within 1e-6 tolerance).