Detect Changes like Humans: Incorporating Semantic Priors for Improved Change Detection
> Yuhang Gan, Wenjie Xuan, Zhiming Luo, Lei Fang, Zengmao Wang, Juhua Liu, Bo Du
This is the PyTorch implementation for Semantic-Aware Change Detection network, namely SA-CDNet, which transfers the common knowledge of the visual foundation models to change detection. Moreover, a single-temporal semantic pre-training strategy is proposed to enrich the semantic prior of the network.
**[Under Review]**
## :fire: News
- **[2025/01/13]**: Codes for training and evaluation are released. This repo is an updated version of [SA-CD](https://github.com/thislzm/SA-CD) by the co-author Zhiming Luo.
## :round_pushpin: Todo
- [x] Release training and inference codes. Instructions on dataset preparation and checkpoints are also provided.
## :sparkles: Highlight
- **A semantic-aware change detection network, namely SA-CDNet.** We propose an advanced SA-CDNet for change detection, which inherits the rich knowledge of natural images from the vision foundation model and adapts these semantic priors to remote sensing change detection. Additionally, a dual-stream feature decoder is derived to mimic the human visual paradigm, which integrates semantic-aware and difference-aware features for robust predictions.
- **A well-designed single-temporal semantic pre-training strategy.** To relieve the scarcity of annotated change detection data, we construct pseudo-change detection data from remote sensing semantic segmentation datasets to pre-train the network. Based on our SA-CDNet, an extra segmentation branch is introduced for improved semantic understanding through discriminating landscapes in single-temporal images.
- **Extensive experiments on five challenging benchmarks. ** We conducted experiments on five benchmarks including LEVIR-CD, LEVIR-CD+, S2Looking, WHU-CD for building changes, and the WHU Cultivated Land for farmland changes. Our method achieves SOTA performance on all datasets. Comprehensive ablation studies proved the effectiveness of our model and pre-training strategy.
## :memo: Introduction
- The architecture of our proposed SA-CDNet, which comprises of a vision foundation model-based feature encoder with change detection adapter, and a dual-stream feature decoder to combine semantic-aware and difference-aware features for robust change detection.
- Our single-temporal semantic pre-training strategy, where we construct pseudo-change detection data from remote sensing semantic segmentation datasets as the pre-training dataset, and an additional single-temporal semantic segmentation task is introduced into the pre-training to improve the semantic understanding of bi-temporal images.
## :hammer_and_wrench: Install
**Recommended**: `Python=3.7` `torch=1.9.0` `CUDA=11.7`
```shell
# set up repository
git https://github.com/DREAMXFAR/SA-CDNet.git
cd SA-CDNet-master
# install conda environment
conda create -n sacdnet python=3.7
conda activate sacdnet
pip install -r requirements.txt
```
## :pushpin: Checkpoints
You can download the following checkpoints and put them in `checkpoints/` and `fastsam_model/`.
| Model | Baidu Yun | Notations |
| ---------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
| FastSAM | [link](https://pan.baidu.com/s/1lS3gPNU6FkTY3uFEAj_ITA?pwd=lzms) | Pre-trained FastSAM models. |
| SA-CDNet (pre-trained) | [link](https://pan.baidu.com/s/1otiGilbOmgzeATVerx7LSQ?pwd=lzms) | Our SA-CDNet pre-trained on pseudo-change data. |
| SA-CDNet (fine-tuned) | [link](https://pan.baidu.com/s/1JrX2jMrJIQJniEg2w0-sog?pwd=lzms) | Our final SA-CDNet after fine-tuning on change detection data. |
## :rocket: Getting Started
> Here we suppose you are in `SA-CDNet-master/`
#### File Structure
- The whole project is supposed to organized as follows.
```shell
# directory
SA-CD
├── checkpoints # model checkpoints, need to download model checkpoints
│ ├── SANet
│ │ ├── LEVIR
│ │ │ ├── no_pretrain.pth
│ │ │ └── with_pretrain.pth
│ │ ├── LEVIR+
│ │ │ ├── no_pretrain.pth
│ │ │ └── with_pretrain.pth
│ │ ├── S2Looking
│ │ │ ├── no_pretrain.pth
│ │ │ └── with_pretrain.pth
│ │ ├── WHU
│ │ │ ├── no_pretrain.pth
│ │ │ └── with_pretrain.pth
│ │ └── WHU_CUL
│ │ ├── no_pretrain.pth
│ │ └── with_pretrain.pth
│ └── pre_train # pre-trained models
│ ├── Building
│ │ └── pretrain_Building.pth
│ └── Cultivalted land
│ └── pretrain_cultivalted.pth
├── datasets # data loader
│ ├── Levir_CD.py
│ ├── Levir_CD2.py
│ ├── data_utils.py
│ ├── data_utils2.py
│ ├── pre_CD.py
│ ├── pre_CD2.py
│ └── pre_CD3.py
├── fastsam_model # checkpoints of vision foundation models, also need to download
│ ├── FastSAM-s.pt
│ └── FastSAM-x.pt
├── models # model definitions
│ ├── FastSAM
│ ├── SAM_Fusion2.py
│ └── preSAM_SNUNet4.py
├── pretrain.py
├── train.py
├── eval.py
├── ultralytics
└── utils
```
#### Date Preparation
1. Download the datasets first. Here we release all the datasets we used for pre-training and fine-tuning. Specifically, the single temporal semantic segmentation datasets are used to construct pseudo-change data for pre-training and change detection datasets are used for fine-tuning.
| Dataset | Baidu Yun | Dataset Category |
| ---------------------------- | ------------------------------------------------------------ | ------------------------------------ |
| `LEVIR` | [link](https://pan.baidu.com/s/11W6Obog_mQm6kXfwfpuKDQ?pwd=lzms) | change detection |
| `LEVIR+` | [link](https://pan.baidu.com/s/1IRfqsIka7PpHEiGWkTh01Q?pwd=lzms) | change detection |
| `S2Looking` | [link](https://pan.baidu.com/s/1UT1weZqiUgjdg68yuAgSJQ?pwd=lzms) | change detection |
| `WHU-CD` | [link](https://pan.baidu.com/s/1g2y2wYnx7OIjIkARu5eAfg?pwd=lzms) | change detection |
| `WHU Cultivate Land Dataset` | [link](https://pan.baidu.com/s/1lIooiYua3wwe4ec-9Zs7Wg?pwd=lzms) | change detection |
| `WHU-Building` | [link](https://pan.baidu.com/s/1qnrWS-0UfVlsSJpPpAodIQ?pwd=lzms) | remote-sensing semantic segmentation |
| `INRIA-Building` | [link](https://pan.baidu.com/s/1f1Q1PnjEQp3PtxlkCNFTgg?pwd=lzms) | remote-sensing semantic segmentation |
| `LoveDA` | [link](https://aistudio.baidu.com/datasetdetail/55681) | remote-sensing semantic segmentation |
| `DLCCC` | [link](https://aistudio.baidu.com/datasetdetail/55681) | remote-sensing semantic segmentation |
| `AIRS` | [link](https://aistudio.baidu.com/datasetdetail/74274) | remote-sensing semantic segmentation |
#### Pre-training
1. Configure the dataset path in the following files.
```shell
# for single dataset, in ./datasets/pre_CD.py, line 16
root = "path/to/pretraining/dataset"
# for two datasets, in ./datasets/pre_CD2.py, line 16, 17
root1 = "/path/to/the/first/dataset"
root2 = "/path/to/the/second/dataset"
# for three datasets, in ./datasets/pre_CD3.py, line 16
root1 = "/path/to/first/dataset"
root2 = "/path/to/second/dataset"
root3 = "/path/to/third/dataset"
```
2. Update the model path of FastSAM in the following files.
```shell
# ./models/preSAM_SUNNet4.py, line 235
model_name: str = '/path/to/fastsam_model/FastSAM-x.pt'
# ./models/SAM_Fusion2.py, line 160
model_name: str = '/path/to/fastsam_model/FastSAM-x.pt'
```
3. Update the directory path of save models in the following file.
```shell
# ./pretrain.py, line 22 and 26
NET_NAME = '/path/of/model/name'
DATA_NAME = '/path/of/data/name'
# more detailed pre-training settings like lr are listed in line 29.
```
#### Fine-tuning
1. Update the path of datasets in the following file.
```shell
# ./datasets/Levir_CD.py, line 16
root = '/path/of/dataset' # data path
```
2. Configure the pathes and parameters for training.
```shell
# the parameter settings for training, in ./train.py
# the path of saved models, in line 20 and 24
NET_NAME = '/path/of/model/name'
DATA_NAME = '/path/of/data/name'
# enable to load pre-trained models, line 40
'load_premodel': True
# the path of pre-trained models, line 43
'chkpt_path': '/path/to/pretrain/model', # pretrain model path
# more detailed training settings like lr are listed in line 26.
```
3. Configure the validation settings.
```shell
# the parameter settings for validation, in ./eval.py
# the model save path, in line 19 and 22
NET_NAME = '/path/of/model/name'
DATA_NAME = '/path/of/data/name'
# the model name for validation, in line 38, 42
'chkpt_path': '/path/to/ckpt',
'load_path': '/path/to/load/data',
```
4. To inference, you can use the validation codes.
## :chart_with_downwards_trend: Performance
1. **Quantitative results.** The performance of the proposed method is evaluated with and without pretraining on multiple public datasets. Results are presented in terms of Precision (P), Recall (R), and F1-Score (F1).
| Datasets |
Methods |
P(%) |
R(%) |
F1(%) |
| LEVIR |
SA - CD w/o pretrain |
92.69 |
89.66 |
91.15 |
| SA - CD w/ pretrain |
91.77 |
91.28 |
91.53 |
| LEVIR + |
SA - CD w/o pretrain |
86.88 |
79.15 |
82.83 |
| SA - CD w/ pretrain |
85.55 |
83.44 |
84.43 |
| S2Looking |
SA - CD w/o pretrain |
81.25 |
54.73 |
65.40 |
| SA - CD w/ pretrain |
81.28 |
56.24 |
66.48 |
| WHU |
SA - CD w/o pretrain |
94.66 |
91.22 |
92.91 |
| SA - CD w/ pretrain |
95.29 |
93.67 |
94.47 |
| WHU Cultivate Land |
SA - CD w/o pretrain |
72.65 |
73.93 |
73.28 |
| SA - CD w/ pretrain |
77.74 |
72.82 |
75.20 |
2. **Qualitative results.** Visualized comparisons with existing SOTA methods on five benchmark change detection datasets.
## :bulb: FAQs
- [x] None
## 💗 Acknowledgements
- Our implementation is greatly based on the [FastSAM](https://github.com/CASIA-IVA-Lab/FastSAM), [SAM-CD](https://github.com/DingLei14/SAM-CD), Thanks for their wonderful works and all contributors.
## :black_nib: Citation
If you find our findings helpful in your research, please consider giving this repository a :star: and citing:
```bibtex
@misc{gan2024detectchangeslikehumans,
title={Detect Changes like Humans: Incorporating Semantic Priors for Improved Change Detection},
author={Yuhang Gan and Wenjie Xuan and Zhiming Luo and Lei Fang and Zengmao Wang and Juhua Liu and Bo Du},
year={2024},
eprint={2412.16918},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.16918},
}
```