Segment Anything Model (SAM): SAM segmentation model

What is the Segment Anything Model?

Segment Anything Model (SAM) is Meta AI's image segmentation model that turns a point, box, or full-image sweep into pixel-accurate object masks. SamPredictor takes one prompt and returns a mask in a single forward pass, while SamAutomaticMaskGenerator masks every object in an image with no prompt needed. It ships as inference code and three pretrained ViT checkpoints, not a training recipe.

Key capabilities: prompts and automatic mask generation

• ✓Two ways to get masks: SamPredictor for a single point/box prompt, SamAutomaticMaskGenerator for every object in the image at once.
• ✓Three ViT backbone checkpoints — vit_h (default), vit_l, and vit_b — swap the model_type string to trade accuracy for speed.
• ✓Trained on a dataset of 11 million images and 1.1 billion masks, which is what gives it strong zero-shot performance on unfamiliar images.
• ✓Mask decoder exports to ONNX, so the exported model can run outside PyTorch — including in a browser, per the demo/ React app.
• ✓CLI entry point (scripts/amg.py) for batch-generating masks over a folder of images without writing Python.
• ✓Output masks come as COCO RLE segmentation plus bbox, area, predicted_iou, and stability_score fields, ready for downstream COCO tooling.

Where SAM fits in computer vision pipelines

• •Pulling object masks for a labeling pipeline instead of hand-drawing polygons.
• •Prepping training data for a downstream detector or segmentation model that needs masks it doesn't have to hand-annotate.
• •Interactive image editors where a user clicks a point and the app needs a mask back immediately (this is what the ONNX web demo shows).
• •Batch-masking a folder of images from the command line via scripts/amg.py, without writing any Python.

Installing SAM and its dependencies

Requires python>=3.8, pytorch>=1.7, and torchvision>=0.8 — install PyTorch/TorchVision with CUDA support first, since the README calls that strongly recommended. Then either `pip install git+https://github.com/facebookresearch/segment-anything.git` or clone the repo and run `pip install -e .`. For mask post-processing, COCO-format export, the example notebooks, and ONNX export, also install `pip install opencv-python pycocotools matplotlib onnxruntime onnx` — none of that is bundled by default. You still need to separately download a model checkpoint (a vit_h/vit_l/vit_b .pth file) before anything runs; the pip package ships code only, not weights.

Running SAM: prompts, automatic masks, and CLI

Point-and-box prompting goes through SamPredictor: load a checkpoint with `sam_model_registry["<model_type>"](checkpoint=...)`, call `predictor.set_image(<your_image>)` once, then `predictor.predict(<input_prompts>)` per prompt. For unattended, whole-image masking, hand the same registry-loaded model to `SamAutomaticMaskGenerator(sam)` and call `.generate(<your_image>)` — no prompts required. The same job also runs from a terminal: `python scripts/amg.py --checkpoint <path> --model-type <model_type> --input <image_or_folder> --output <path>`. To get an ONNX file for browser or edge use, run `python scripts/export_onnx_model.py --checkpoint <path> --model-type <model_type> --output <path>`; the README recommends the latest stable PyTorch for that export step specifically.

How SAM compares to other segmentation approaches

Frequently asked questions about SAM

Who should try it — and who should skip

Try it if you're building a labeling or annotation pipeline and want masks without hand-drawing polygons, or you're prototyping a click-to-mask feature and can run inference on a GPU. Skip it if your images already suit simple thresholding or contour detection — SAM is overkill there — or if you actually need video, in which case go straight to SAM 2 instead of stretching this repo to do it.