UniCodec (ACL 2025 Main)

UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook<br> Yidi Jiang,Qian Chen,Shengpeng Ji,Yu Xi,Wen Wang,Chong Zhang,Xianghu Yue,Shiliang Zhang,Haizhou Li<br> National University of Singapore; Tongyi Speech Lab<br>

In this work, we introduce UniCodec, a unified audio codec with a single codebook to support multi-domain audio data, including speech, music, and sound.

To achieve this, we propose a partitioned domain-adaptive codebook method with domain Mixture-of-Experts strategy to capture the distinct characteristics of each audio domain. Furthermore, to enrich the semantic density of the codec without auxiliary modules, we propose a self-supervised mask prediction modeling approach.

<div align=center> <img src="https://github.com/Jiang-Yidi/UniCodec/blob/main/overview.png" width="50%"> </div>

As a single unified codec model, UniCodec achieves superior subjective reconstruction performance while maintaining a high compression rate in all three domains (speech/music/sound).

Installation

conda create -n unicodec python=3.9
conda activate unicodec
pip install -r requirements.txt

Train

python train.py fit --config ./configs/xxx.yaml

Infer

Model checkpoint 🤗 is available in Huggingface.

python infer_audio.py

Citation

@article{jiang2025unicodec,
  title={UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook},
  author={Jiang, Yidi and Chen, Qian and Ji, Shengpeng and Xi, Yu and Wang, Wen and Zhang, Chong and Yue, Xianghu and Zhang, ShiLiang and Li, Haizhou},
  journal={arXiv preprint arXiv:2502.20067},
  year={2025}
}