PointGST
[IEEE TPAMI] Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning
Install / Use
/learn @jerryfeng2003/PointGSTREADME
Dingkang Liang<sup>1</sup>* ,Tianrui Feng<sup>1</sup>* ,Xin Zhou<sup>1</sup>* , Yumeng Zhang<sup>2</sup>, Zhikang Zou<sup>2</sup>, and Xiang Bai<sup> 1✉️</sup>
<sup>1</sup> Huazhong University of Science and Technology, <sup>2</sup> Baidu Inc.
(*) equal contribution, (✉️) corresponding author.
News
[2025-07-26] PointGST is accepted by TPAMI. 🎉
[2024-10-10] PointGST is released. 🔥
Abstract
Recently, leveraging pre-training techniques to enhance point cloud models has become a hot research topic. However, existing approaches typically require full fine-tuning of pre-trained models to achieve satisfied performance on downstream tasks, accompanying storage-intensive and computationally demanding. To address this issue, we propose a novel Parameter-Efficient Fine-Tuning (PEFT) method for point cloud, called PointGST (Point cloud Graph Spectral Tuning). PointGST freezes the pre-trained model and introduces a lightweight, trainable Point Cloud Spectral Adapter (PCSA) to fine-tune parameters in the spectral domain.
<div align="center"> <img src="./figure/intro.png" width = "888" align=center /> </div>Extensive experiments on challenging point cloud datasets across various tasks demonstrate that PointGST not only outperforms its fully fine-tuning counterpart but also significantly reduces trainable parameters, making it a promising solution for efficient point cloud learning. More importantly, it improves upon a solid baseline by +2.28%, 1.16%, and 2.78%, resulting in 99.48%, 97.76%, and 96.18% on the ScanObjNN OBJ_BG, OBJ_OBLY, and PB_T50_RS datasets, respectively. This advancement establishes a new state-of-the-art, using only 0.67% of the trainable parameters.
Overview
<div align="center"> <img src="./figure/pipeline.png" width = "888" align=center /> </div>Getting Started
Installation
We recommend using Anaconda for the installation process:
git clone https://github.com/jerryfeng2003/PointGST.git
cd PointGST/
Requirements
conda create -y -n pgst python=3.9
conda activate pgst
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
# Chamfer Distance & emd
cd ./extensions/chamfer_dist
python setup.py install --user
cd ../emd
python setup.py install --user
# PointNet++
pip install "git+https://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"
# GPU kNN
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
Datasets
See DATASET.md for details.
Main Results
<div align="center"> <img src="./figure/result.png" width = "888" align=center /> </div> <div align="center"> <img src="./figure/result2.png" width = "888" align=center /> </div>| Baseline | Trainable Parameters | Dataset | Config | Acc. | Download | | :---- | :---- | :---- | :---- | :---- | :---- | | Point-MAE <br> (ECCV 22) | 0.6M |ModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RS | modelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest | 93.5 <br> 91.74 <br> 90.19 <br> 85.29 | ckpt <br> ckpt <br> ckpt <br> ckpt | | ACT <br> (ICLR 23)| 0.6M |ModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RS | modelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest | 93.4 <br> 93.46 <br> 92.60 <br> 88.27 | ckpt <br> ckpt <br> ckpt <br> ckpt | | ReCon <br> (ICML 23) | 0.6M |ModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RS | modelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest | 93.6 <br> 94.49 <br> 92.94 <br> 89.49 | ckpt <br> ckpt <br> ckpt <br> ckpt | | PointGPT-L <br> (NeurIPS 24)| 2.4M |ModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RS | modelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest | 94.8 <br> 98.97 <br> 97.59 <br> 94.83| ckpt <br> ckpt <br> ckpt <br> ckpt | | PointGPT-L (voting) <br> (NeurIPS 24) | 2.4M |ModelNet40 <br> OBJ_BG <br> OBJ_ONLY <br> PB_T50_RS | modelnet <br> scan_objbg <br> scan_objonly <br> scan_hardest | 95.3 <br> 99.48 <br> 97.76 <br> 96.18| log <br> log <br> log <br> log |
The evaluation commands with checkpoints should be in the following format:
CUDA_VISIBLE_DEVICES=<GPU> python main.py --test --config <path/to/cfg> --exp_name <path/to/output> --ckpts <namet>
# further enable voting mechanism
CUDA_VISIBLE_DEVICES=<GPU> python main.py --test --vote --config <path/to/cfg> --exp_name <path/to/output> --ckpts <name>
All the experiments are conducted on a single NVIDIA 3090 GPU.
t-SNE visualization
# t-SNE on ScanObjectNN
CUDA_VISIBLE_DEVICES=<GPU> python main.py --config <path/to/cfg> --ckpts <path/to/ckpt> --tsne --exp_name <name>
Training
If you plan to fine-tune on top of pretrained models, please download the weights for Point-MAE, ACT, ReCon, or PointGPT accordingly.
CUDA_VISIBLE_DEVICES=<GPU> python main.py --finetune_model --config <path/to/cfg> --ckpts <path/to/ckpt> --exp_name <name>
To Do
- [x] Release the inference code for classification.
- [x] Release the checkpoints for classification.
- [x] Release the training code for classification.
Acknowledgement
This project is based on Point-BERT (paper, code), Point-MAE (paper, code), ACT(paper, code), ReCon (paper, code), PointGPT(paper, code), IDPT (paper, code), and DAPT(paper, code). Thanks for their wonderful works.
Citation
If you find this repository useful in your research, please consider giving a star ⭐ and a citation.
@article{liang2024pointgst,
title={Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learnin
