<div align="center"> <h1><img src="images/CrossEarth.png" width="5%"> CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation</h1>

Ziyang Gong ^1∗, Zhixiang Wei ^2∗, Di Wang ^3∗, Xiaoxing Hu^8∗, Xianzheng Ma³, Hongruixuan Chen ^4,5, Yuru Jia ^6,7, Yupeng Deng ⁹, Zhenming Ji^10†, Xiangwei Zhu^8†, Xue Yang^1†, Naoto Yokoya^4,5, Jing Zhang³, Bo Du³, Junchi Yan¹, Liangpei Zhang³

¹ SJTU ² USTC ³ WHU ⁴ U Tokyo ⁵ RIKEN ⁶ KU Leuven ⁷ KTH ⁸ BIT ⁹ SYSU ¹⁰ UCAS

^* Equal contribution. ^† Corresponding author.

<img src="https://visitor-badge.laobi.icu/badge?page_id=Cuzyoung.CrossEarth&left_color=%2363C7E6&right_color=%23CEE75F"> <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg"> <img src="https://img.shields.io/github/stars/Cuzyoung/CrossEarth.svg?logo=github&label=Stars&color=white">

<a href="https://cuzyoung.github.io/CrossEarth-Homepage/"><img src="https://img.shields.io/badge/🌍_Project_Page-E4F578?style=for-the-badge" style="width: 113px;"></a> <a href="https://github.com/Cuzyoung/CrossEarth/"><img src="https://img.shields.io/badge/WeChat-07C160?style=for-the-badge&logo=wechat&logoColor=white" style="width: 80px;"></a> <a href="https://www.linkedin.com/in/ziyang-gong-382a182b1/"><img src="https://img.shields.io/badge/linkedin-%230077B5.svg?style=for-the-badge&logo=linkedin&logoColor=white" style="width: 75px;"></a> <a href="https://pan.baidu.com/s/1nOQlZUTeySn6-jiqjheWhQ?pwd=eq40"><img src="https://img.shields.io/badge/Baidu Netdisk-63C7E6.svg?style=for-the-badge&logo=Baidu&logoColor=white" style="width: 123px;"></a> <a href="https://huggingface.co/Cusyoung/CrossEarth/tree/main"> <img src="https://img.shields.io/badge/🤗_Hugging_Face-FFD21E?style=for-the-badge" style="width: 115px;"></a>

<img src="images/teaser_new.png" width="100%"> </div> <br> <br>

🔥🔥🔥 News

• [2024/11/14] We are releasing the benchmark collection. Click here to get illustrations of benchmarks!

• [2024/11/06] The most checkpoints have been uploaded and you can access them in the huggingface badges.

• The environment and inference steps please refer to the following installation. The inference codes and weights will be coming soon.

• The benchmark collection in the paper is releasing and you can access it at here.

• 🎉🎉🎉 CrossEarth is the first VFM for Remote Sensing Domain Generalization (RSDG) semantic segmentation. We just release the arxiv paper of CrossEarth. You can access CrossEarth at here.

📑 Table of Content

• Visualization
• Environment Requirements
• Inference steps
• Training steps
• Model Weights with Configs

<!-- - [Citation](#citation) -->

Visualization

In Radar figure:

• CrossEarth achieves SOTA performances on 23 evaluation benchmarks across various segmentation scenes, demonstrating strong generalizability.

In UMAP figures:

• CrossEarth extracts features that cluster closely for the same class across different domains, forming well-defined groups in feature space, demonstrating its ability to learn robust, domain-invariant features.

• Moreover, CrossEarth features exhibit high inter-class separability, forming unique clusters for each class and underscoring its strong representational ability to distingguish different categories.

<img src="images/visual.png" width="100%">

Environment Requirements:

conda create -n CrossEarth -y
conda activate CrossEarth
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia -y
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
pip install "mmsegmentation>=1.0.0"
pip install "mmdet>=3.0.0"
pip install xformers=='0.0.20' 
pip install -r requirements.txt
pip install future tensorboard

Inference steps:

First, download the model weights from the huggingface or Baidu Netdisk in the above badges. Notably, the checkpoints of dinov2_converted.pth and dinov2_converted_1024x1024.pth are needed for inference. Please download them and put them in the CrossEarth/checkpoints folder.

Second, change the file path in experiment config files (configs/base/datasets/xxx.py and configs/CrossEarth_dinov2/xxx.py), and run the following command to inference. (Take 512x512 inference as an example)

python tools/test.py configs/CrossEarth_dinov2/CrossEarth_dinov2_mask2former_512x512_bs1x4.py ./checkpoints/xxx.pth

Notably, save path of pseudo labels is in the experiment config file. When testing CrossEarth on different benchmarks, you also need to change the class number in CrossEarth_dinov2_mask2former.py file.

Training steps:

Coming soon.

Model Weights with Configs

| Dataset | Benchmark | Model | Config | Log | |----------|----------|-------| -------| ----| |ISPRS Potsdam and Vaihingen|P(i)2V| Potsdam(i)-source.pth | dg_potIRRG2RGB_512x512.py |-| |- |P(i)2P(r)| Potsdam(i)-source.pth | dg_potIRRG2RGB_512x512.py |-| |- |P(r)2P(i)| Potsdam(r)-source.pth | dg_potRGB2IRRG_512x512.py |-| |- |P(r)2V| Potsdam(r)-source.pth | dg_potRGB2IRRG_512x512.py |- | |- |V2P(i)| Vaihingen-source.pth | dg_vai2potIRRG_512x512.py|- | |- |V2P(r)| Vaihingen-source.pth| dg_vai2potIRRG_512x512.py |- | |LoveDA|Rural2Urban| R2U.pth | dg_loveda_rural2urban_1024x1024.py |Coming Soon | |- |Urban2Rural| U2R.pth | dg_loveda_urban2rural_1024x1024.py |(Coming Soon) | |WHU Building|A2S| WHU-Building-A2S.pth | dg_building_aerial2satellite.py|Coming Soon | |- |S2A| WHU-Building-S2A.pth| dg_building_satellite2aerial.py |Coming Soon | |DeepGlobe and Massachusetts |D2M| D2M-Rein/D2M-MTP | dg_deepglobe2massachusetts_1024x1024.py |Coming Soon | |ISPRS Potsdam and RescueNet|P(r)2Res| Potsdam(r)2RescueNet.pth |dg_potsdamIRRG2rescue_512x512.py |Coming Soon | |- |P(i)2Res| Potsdam(i)2RescueNet.pth | dg_potsdamRGB2rescue_512x512.py |Coming Soon | |CASID|Sub2Sub| casid-sub-source.pth | dg_casid_subms_1024x1024|Coming Soon | |- |Sub2Tem|casid-sub-source.pth | dg_casid_subms_1024x1024 |- | |- |Sub2Tms| casid-sub-source.pth | dg_casid_subms_1024x1024 |-| |- |Susb2Trf| casid-sub-source.pth| [dg_casid_subms_1024x1024](https://