<div align="center"> <h2> Time Travelling Pixels: Bitemporal Features Integration with Foundation Model for Remote Sensing Image Change Detection </h2> </div> <br>

<br>

<div align="center"> <a href="https://kychen.me/TTP"> <span style="font-size: 20px; ">Project Page</span> </a>      <a href="https://arxiv.org/abs/2312.16202"> <span style="font-size: 20px; ">arXiv</span> </a>      <a href="https://huggingface.co/spaces/KyanChen/TTP"> <span style="font-size: 20px; ">HFSpace</span> </a>      <a href="resources/ttp.pdf"> <span style="font-size: 20px; ">PDF</span> </a> </div> <br> <br>

English | 简体中文

</div>

Introduction

The repository is the code implementation of the paper Time Travelling Pixels: Bitemporal Features Integration with Foundation Model for Remote Sensing Image Change Detection, based on MMSegmentation and Open-CD projects.

The current branch has been tested under PyTorch 2.x and CUDA 12.1, supports Python 3.7+, and is compatible with most CUDA versions.

If you find this project helpful, please give us a star ⭐️, your support is our greatest motivation.

<details open> <summary>Main Features</summary>

Consistent API interface and usage with MMSegmentation
Open sourced the TTP model in the paper
Tested with AMP training method
Supports multiple dataset extensions

</details>

Update Log

🌟 2023.12.23 Released the TTP project code, which is completely consistent with the API interface and usage of MMSegmentation.

🌟 2023.12.30 Released the model trained on Levir-CD.

🌟 2024.02.10 This project has been included in the Open-CD project.

Introduction
Update Log
Table of Contents
Installation
Dataset Preparation
Model Training
Model Testing
Image Prediction
FAQ
Acknowledgements
Citation
License
Contact Us

Installation

Dependencies

Linux or Windows
Python 3.7+, recommended 3.10
PyTorch 2.0 or higher, recommended 2.1
CUDA 11.7 or higher, recommended 12.1
MMCV 2.0 or higher, recommended 2.1

Environment Installation

We recommend using Miniconda for installation. The following command will create a virtual environment named ttp and install PyTorch and MMCV.

Note: If you have experience with PyTorch and have already installed it, you can skip to the next section. Otherwise, you can follow these steps to prepare.

Step 0: Install Miniconda.

Step 1: Create a virtual environment named ttp and activate it.

conda create -n ttp python=3.10 -y
conda activate ttp

Step 2: Install PyTorch2.1.x.

Linux/Windows:

pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121

conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia

Step 3: Install MMCV2.1.x.

pip install -U openmim
mim install mmcv==2.1.0

Step 4: Install other dependencies.

pip install -U wandb einops importlib peft==0.8.2 scipy ftfy prettytable torchmetrics==1.3.1 transformers==4.38.1

</details>

Install TTP

Download or clone the TTP repository.

git clone git@github.com:KyanChen/TTP.git
cd TTP

Dataset Preparation

Levir-CD Change Detection Dataset

Dataset Download

Image and label download address: Levir-CD.

Organization Method

You can also choose other sources to download the data, but you need to organize the dataset in the following format:

${DATASET_ROOT} # Dataset root directory, for example: /home/username/data/levir-cd
├── train
│   ├── A
│   ├── B
│   └── label
├── val
│   ├── A
│   ├── B
│   └── label
└── test
    ├── A
    ├── B
    └── label

Note: In the project folder, we provide a folder named data, which contains an example of the organization method of the above dataset.

Other Datasets

If you want to use other datasets, you can refer to MMSegmentation documentation to prepare the datasets.

</details>

Model Training

TTP Model

Config File and Main Parameter Parsing

We provide the configuration files of the TTP model used in the paper, which can be found in the configs/TTP folder. The Config file is completely consistent with the API interface and usage of MMSegmentation. Below we provide an analysis of some of the main parameters. If you want to know more about the meaning of the parameters, you can refer to MMSegmentation documentation.

Parameter Parsing:

work_dir: The output path of the model training, which generally does not need to be modified.
default_hooks-CheckpointHook: Checkpoint saving configuration during model training, which generally does not need to be modified.
default_hooks-visualization: Visualization configuration during model training, comment out during training and uncomment during testing.
vis_backends-WandbVisBackend: Configuration of network-side visualization tools, after opening the comment, you need to register an account on the wandb official website, and you can view the visualization results during the training process in the network browser.
sam_pretrain_ckpt_path: The checkpoint path of the SAM backbone provided by MMPretrain, refer to download address.
model-backbone-peft_cfg: Whether to introduce fine-tuning parameters, which generally does not need to be modified.
dataset_type: The type of dataset, needs to be modified according to the type of dataset.
data_root: Dataset root directory, modify to the absolute path of the dataset root directory.
batch_size_per_gpu: The batch size of a single card, needs to be modified according to the memory size.
resume: Whether to resume training, which generally does not need to be modified.
load_from: The checkpoint path of the model's pre-training, which generally does not need to be modified.
max_epochs: The maximum number of training rounds, which generally does not need to be modified.

</details>

Single Card Training

python tools/train.py configs/TTP/xxx.py  # xxx.py is the configuration file you want to use

Multi-card Training

sh ./tools/dist_train.sh configs/TTP/xxx.py ${GPU_NUM}  # xxx.py is the configuration file you want to use, GPU_NUM is the number of GPUs used

Other Instance Segmentation Models

If you want to use other change detection models, you can refer to Open-CD to train the models, or you can put their Config files into the configs folder of this project, and then train them according to the above method.

</details>

Model Testing

Single Card Testing:

python tools/test.py configs/TTP/xxx.py ${CHECKPOINT_FILE}  # xxx.py is the configuration file you want to use, CHECKPOINT_FILE is the checkpoint file you want to use

Multi-card Testing:

sh ./tools/dist_test.sh configs/TTP/xxx.py ${CHECKPOINT_FILE} ${GPU_NUM}  # xxx.py is the configuration file you want to use, CHECKPOINT_FILE is the checkpoint file you want to use, GPU_NUM is the number of GPUs used

Note: If you need to get the visualization results, you can uncomment default_hooks-visualization in the Config file.

Image Prediction

Single Image Prediction:

python demo/image_demo_with_cdinferencer.py ${IMAGE_FILE1} ${IMAGE_FILE2} configs/TTP/ttp_sam_large_levircd_infer.py --checkpoint ${CHECKPOINT_FILE} --out-dir ${OUTPUT_DIR}  # IMAGE_FILE is the image file you want to predict, xxx.py is the configuration file, CHECKPOINT_FILE is the checkpoint file you want to use, OUTPUT_DIR is the output path of the prediction result

FAQ

We have listed some common problems and their corresponding solutions here. If you find that some problems are missing, please feel free to provide a PR to enrich this list. If you cannot get help here, please use issue to seek help. Please fill in all the required information in the template, which will help us locate the problem faster.

1. Do I need to install MMSegmentation, MMPretrain, MMDet, Open-CD?

We recommend that you do not install them, because we have partially modified their code, which may cause errors in the code if you install them. If you get an error that

TTP

Install / Use

README

Introduction

Update Log

Table of Contents

Installation

Dependencies

Environment Installation

Install TTP

Dataset Preparation

Levir-CD Change Detection Dataset

Dataset Download

Organization Method

Other Datasets

Model Training

TTP Model

Config File and Main Parameter Parsing

Single Card Training

Multi-card Training

Other Instance Segmentation Models

Model Testing

Single Card Testing:

Multi-card Testing:

Image Prediction

Single Image Prediction:

FAQ

1. Do I need to install MMSegmentation, MMPretrain, MMDet, Open-CD?