Grounded-SAM-2
Grounded SAM 2: Ground and Track Anything with Grounding DINO, Grounding DINO 1.5 and SAM 2
Contents
Installation
Since we need the CUDA compilation environment to compile the Deformable Attention operator used in Grounding DINO, we need to check whether the CUDA environment variables have been set correctly (which you can refer to Grounding DINO Installation for more details). You can set the environment variable manually as follows if you want to build a local GPU environment for Grounding DINO to run Grounded SAM 2:
export CUDA_HOME=/path/to/cuda-12.1/
Install Segment Anything 2:
pip install -e .
Install Grounding DINO:
pip install --no-build-isolation -e grounding_dino
Downgrade the version of the supervision library to 0.6.0 to use its original API for visualization (we will update our code to be compatible with the latest version of supervision in the future release):
pip install supervision==0.6.0
Download the pretrained SAM 2 checkpoints:
cd checkpoints
bash download_ckpts.sh
Download the pretrained Grounding DINO checkpoints:
cd gdino_checkpoints
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth
Grounded-SAM-2 Demo
Grounded-SAM-2 Image Demo (with Grounding DINO)
Note that Grounding DINO has already been supported in Huggingface, so we provide two choices for running Grounded-SAM-2 model:
- Use huggingface API to inference Grounding DINO (which is simple and clear)
python grounded_sam2_hf_model_demo.py
- Load local pretrained Grounding DINO checkpoint and inference with Grounding DINO original API (make sure you've already downloaded the pretrained checkpoint)
python grounded_sam2_local_demo.py
Grounded-SAM-2 Image Demo (with Grounding DINO 1.5 & 1.6)
We've already released our most capable open-set detection model Grounding DINO 1.5 & 1.6, which can be combined with SAM 2 for stronger open-set detection and segmentation capability. You can apply the API token first and run Grounded-SAM-2 with Grounding DINO 1.5 as follows:
Install the latest DDS cloudapi:
pip install dds-cloudapi-sdk
Apply your API token from our official website here: request API token.
python grounded_sam2_gd1.5_demo.py