🏥 Medical-SAM3
A Foundation Model for Universal Prompt-Driven Medical Image Segmentation
```bash
cd inference
python run_medsam3_evaluation.py --checkpoint /path/to/checkpoint.pt --model-name medsam3
```
SAM3 is bundled in this repo (`sam3/`); no separate clone required.
## 🎬 3D inference, training & evaluation
Run text-prompted inference on a single NPZ volume or a directory. The current
3D policy covers the full depth in independent four-slice clips and does not
use ground-truth masks, boxes, points, or ground-truth-selected slices.
```bash
python medical/inference_3d.py \
--input /path/to/case_or_directory \
--output-dir outputs/3d \
--checkpoint /path/to/medical_sam3_3d.pt \
--prompt '1=liver'
```
Outputs are compressed NPZ files containing `segs` with the original `(D,H,W)`
geometry. Prompts can also come from each input's `text_prompts` dictionary or
a per-case JSON file.
### Training & benchmark evaluation
Fine-tune SAM3 on **3D** annotations and run held-out **3D / JSON** eval (train-aligned protocol).
```bash
export MEDSAM3_ROOT=$(pwd)
export MEDSAM3_DATA_ROOT=/path/to/project
export MEDSAM3_CVPR_ROOT=$MEDSAM3_DATA_ROOT/converted_cvpr_biomedsegfm
pip install -e ".[train]"
bash medical/training/prepare_config_paths.sh # once, if yaml paths differ
# Train
python sam3/train/train.py -c configs/medsam3_stage1_train_all_unified --num-gpus 4
# Eval
python medical/evaluation/3d_val.py \
--ckpt experiments/medsam3_stage1_train_all_unified/checkpoints/checkpoint.pt \
--out-dir experiments/results/3d_val/unified \
--eval-all-volumes --text-only --use-train-prompts
```
| Path | Role |
|------|------|
| `inference/` | 2D image inference & public benchmark eval |
| `medical/` | 3D **training** + **3D/JSON evaluation** |
| `sam3/` | SAM3 model, trainer, Hydra config |
| `assets/` | BPE vocabulary |
| `experiments/` | Checkpoints & eval outputs (runtime) |
## 📅 Todo List
| Feature | Status | Description |
| :--- | :---: | :--- |
| **Demo** | 🚧 Doing | Online interactive demo. |
| **Data Scaling** | 🚧 Doing | Expand training corpus and benchmarks. |
| **3D Training** | ✅ Released | `medical/` + `medsam3_stage1_train_all_unified` config. |
| **Medical-SAM3 Agent** | 📅 Planned | LLM agentic segmentation. |
## 📝 Citation
```bibtex
@article{jiang2026medicalsam3,
title={Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation},
author={Jiang, Chongcong and Ding, Tianxingjian and Song, Chuhan and Tu, Jiachen and Yan, Ziyang and Shao, Yihua and Wang, Zhenyi and Shang, Yuzhang and Han, Tianyu and Tian, Yu},
journal={arXiv preprint arXiv:2601.10880},
year={2026},
url={https://arxiv.org/abs/2601.10880}
}
```