Uni Sign

<h3 align="center"><a href="" style="color:#9C276A"> Uni-Sign: Toward Unified Sign Language Understanding at Scale</a></h3> <h5 align="center"> If our project helps you, please give us a star🌟 on GitHub, that would motivate us a lot! </h2> <h5 align="center">

</h5>

💥 News

[2025/1/25] This paper is accepted by ICLR 2025 🎉🎉!

[2025/2/24] Release CSL-News dataset and code implementation.

[2025/6/5] Release lighter pose extraction and online inference implementation. Check it out here 🎊🎊.

[2025/12/20] Release pose-only checkpoints and inference code for How2Sign and OpenASL. We hope this will be useful to researchers interested in ASL. Check it out here 👏👏.

🛠️ Installation

We suggest to create a new conda environment.

# create environment
conda create --name Uni-Sign python=3.9
conda activate Uni-Sign
# install other relevant dependencies
pip install -r requirements.txt
# (Optional) If you want to evaluate BLEURT scores on the How2Sign or OpenASL datasets
git clone https://github.com/google-research/bleurt.git
cd bleurt
pip install .
cd ../
wget https://storage.googleapis.com/bleurt-oss-21/BLEURT-20.zip .
unzip BLEURT-20.zip

📖 Preparation

Please follow the instructions provided in DATASET.md for data preparation.

💻 Model Checkpoints

The Uni-Sign checkpoints are available for download Here.

🔨 Training & Evaluation

All scripts must be executed within the Uni-Sign directory.

For the pre-training code, please refer to Issue #15 for detailed instructions. Thanks to @williams-bert for generously sharing the code implementation.

Training

Stage 1: pose-only pre-training.

bash ./script/train_stage1.sh

Stage 2: RGB-pose pre-training.

bash ./script/train_stage2.sh

Stage 3: downstream fine-tuning.

bash ./script/train_stage3.sh

Evaluation

After completing stage 3 fine-tuning, performance evaluation on a single GPU can be performed using the following command:

bash ./script/eval_stage3.sh

🎨 Inference Results

If you'd like to skip the environment setup and model weight configuration, you can find sample pose-only sign language translation results (stage 2) Here for quick comparison.

👨‍💻 Todo

• [x] Release CSL-News dataset
• [x] Release Uni-Sign implementation
• [x] Release pose extraction and online inference implementation

📮 Contact

If you have any questions, please feel free to contact Zecheng Li (lizecheng19@gmail.com). Thank you.

👍 Acknowledgement

The codebase of Uni-Sign is adapted from GFSLT-VLP, while the implementations of the pose/temporal encoders are derived from CoSign. We sincerely appreciate the authors of CoSign for personally sharing their code 🙏.
We are also grateful for the following projects our Uni-Sign arise from:

• 🤟SSVP-SLT: a excellent sign language translation framework!
• 🏃️MMPose: an open-source toolbox for pose estimation.
• 🤠FUNASR: a high-performance speech-to-text toolkit.

📑 Citation

If you find Uni-Sign useful for your research and applications, please cite using this BibTeX:

@article{li2025uni,
  title={Uni-Sign: Toward Unified Sign Language Understanding at Scale},
  author={Li, Zecheng and Zhou, Wengang and Zhao, Weichao and Wu, Kepeng and Hu, Hezhen and Li, Houqiang},
  journal={arXiv preprint arXiv:2501.15187},
  year={2025}
}