Segment Anything Model (SAM): Definition, Applications, and Training Data Requirements

The Segment Anything Model (SAM) is a foundation model for image segmentation released by Meta AI in April 2023. It was designed to be a general-purpose, promptable segmentation system: given an image and a prompt indicating what to segment (a point click, a bounding box, a coarse mask, or a text description), SAM produces a detailed segmentation mask delineating the indicated object or region at pixel precision. The model was trained on SA-1B, a dataset of 1.1 billion masks spanning 11 million images, making it the largest segmentation dataset and model at the time of release.

SAM's architecture separates the expensive image encoding step from the cheap prompt processing and mask generation steps. The image encoder — a Vision Transformer (ViT-H with 632 million parameters) — processes the full image once to produce dense feature embeddings. The prompt encoder maps different prompt types (points, boxes, masks, text) to a common embedding space. The mask decoder — a lightweight transformer with only 4 million parameters — combines image and prompt embeddings to predict segmentation masks. This asymmetric design means that once an image is encoded (~150ms on GPU), any number of prompts can be processed in real time (~50ms each), enabling interactive and iterative segmentation workflows that are impractical with traditional per-query segmentation models.

For physical AI and robotics, SAM addresses a fundamental perception challenge: producing accurate object boundaries for downstream reasoning. Grasp planning requires knowing exactly where an object ends and the supporting surface begins. Navigation requires precise obstacle boundaries for collision avoidance. Manipulation planning needs object masks to estimate shape, pose, and grasp affordances. Before SAM, obtaining these masks required either training task-specific segmentation models (expensive, narrow) or using generic models that underperformed on novel objects. SAM's zero-shot capability — segmenting objects it has never been explicitly trained on — provides a general-purpose solution that works across the diverse objects and environments that robots encounter.

SAM 2 (Ravi et al., 2024) extended the model to video segmentation, adding temporal consistency through a memory attention mechanism that tracks objects across frames. For robotics, this is critical: during a manipulation task, the robot needs to maintain consistent segmentation of the target object as it moves, deforms, or becomes partially occluded. SAM 2 achieves this with a streaming architecture that processes video frames sequentially, propagating object identities through a memory bank of previous frame embeddings. On the SA-V video dataset (50,900 videos with 642,600 masklets), SAM 2 demonstrated state-of-the-art video object segmentation while maintaining real-time performance.

Integrating SAM into a robotics perception pipeline requires decisions about prompt generation, mask selection, and computational budgeting. The choice of prompt type determines segmentation quality and automation level. Point prompts (a single click on the object of interest) are fast to generate programmatically (from detector outputs or user input) but can be ambiguous — a point on a cup might segment just the handle, the cup body, or the entire cup with its contents. Box prompts from an upstream object detector produce more reliable segmentation by constraining the spatial extent. For fully automated pipelines, a common pattern is: run an object detector (DINO, Grounding DINO, or YOLO) to get bounding boxes, then refine each box to a precise mask with SAM.

SAM often produces multiple candidate masks per prompt at different granularity levels (part, object, group). The mask decoder outputs three masks with confidence scores — typically corresponding to sub-part, whole-object, and super-object segmentations. Selecting the right granularity level requires task-specific logic: grasp planning needs the whole-object mask, while dexterous manipulation might need the part-level mask (the cup handle specifically).

Computational cost on robot hardware is a practical concern. The ViT-H image encoder requires ~150ms on an NVIDIA Jetson Orin, which limits the frame rate to ~7 FPS if run on every frame. Strategies to improve throughput include: using the smaller ViT-B variant (3x faster, modest accuracy loss), encoding only keyframes and propagating masks using SAM 2's memory mechanism, or running the encoder asynchronously while the control loop uses the most recent mask. For tabletop manipulation at 1-5 Hz control rates, the standard ViT-H runs comfortably; for mobile navigation at 10+ Hz, the smaller variants or sparse encoding strategies become necessary.

Claru supports SAM-based robotics perception by providing domain-specific data for fine-tuning and evaluation. Our annotated images from robot workspace environments — captured with the same cameras, viewpoints, and lighting that the deployed system will encounter — enable teams to measure and improve SAM's performance on their specific target domain. Annotations include pixel-precise masks in COCO format, object identity labels, and per-mask quality scores, compatible with SAM fine-tuning and standard evaluation pipelines.

Claru provides the domain-specific annotation data that bridges SAM's general zero-shot capabilities with the precision requirements of production robotics. Our 10,000+ annotators produce pixel-precise segmentation masks across diverse real-world environments — kitchens, workshops, retail spaces, outdoor settings — using the same cameras and viewpoints that robot perception systems encounter in deployment. For teams fine-tuning SAM on their target domain, Claru delivers COCO-format mask annotations with object identity labels, covering the specific object categories, surface materials, and lighting conditions where zero-shot performance falls short. For teams evaluating SAM within larger perception pipelines, our annotated test sets provide ground truth for measuring mask IoU, boundary precision, and downstream task impact. With 3M+ annotated clips spanning 100+ cities, Claru enables robotics teams to maximize SAM's potential by closing the gap between internet-photo pretraining and real-world deployment conditions.