:trident: Speech Trident - Awesome Speech LM

<p align="center"> <img src="assets/Speech-Trident-v4.png" alt="Speech Trident" style="width:70%;"> </p>

In this repository, we survey three crucial areas: (1) representation learning, (2) neural codec, and (3) language models that contribute to speech/audio large language models.

1.⚡ Speech Representation Models: These models focus on learning structural speech representations, which can then be quantized into discrete speech tokens, often refer to semantic tokens.

2.⚡ Speech Neural Codec Models: These models are designed to learn speech and audio discrete tokens, often referred to as acoustic tokens, while maintaining reconstruction ability and low bitrate.

3.⚡ Speech Large Language Models: These models are trained on top of speech and acoustic tokens in a language modeling approach. They demonstrate proficiency in tasks on speech understanding and speech generation.

:trident: Contributors

<table> <tr> <td align="center"> <a href="https://kwchang.org/"> <img src="https://avatars.githubusercontent.com/u/20485030?v=4" width="100px;" style="border-radius: 50%;" alt=""/> <br /> _{<b>Kai-Wei Chang</b>} </a> </td> <td align="center"> <a href="https://hbwu-ntu.github.io/"> <img src="https://scholar.googleusercontent.com/citations?view_op=medium_photo&user=-bB-WHEAAAAJ&citpid=1" width="100px;" style="border-radius: 50%;" alt=""/> <br /> _{<b>Haibin Wu</b>} </a> </td> <td align="center"> <a href="https://scholar.google.com.tw/citations?user=-d6aNP0AAAAJ&hl=zh-TW"> <img src="https://scholar.googleusercontent.com/citations?view_op=medium_photo&user=-d6aNP0AAAAJ&citpid=2" width="100px;" style="border-radius: 50%;" alt=""/> <br /> _{<b>Wei-Cheng Tseng</b>} </a> </td> <td align="center"> <a href="https://kehanlu.com/"> <img src="https://avatars.githubusercontent.com/u/22369406?v=4" width="100px;" style="border-radius: 50%;" alt=""/> <br /> _{<b>Kehan Lu</b>} </a> </td> <td align="center"> <a href="https://github.com/kuan2jiu99"> <img src="https://avatars.githubusercontent.com/u/95064418?v=4" width="100px;" style="border-radius: 50%;" alt=""/> <br /> _{<b>Chun-Yi Kuan</b>} </a> </td> <td align="center"> <a href="https://speech.ee.ntu.edu.tw/~hylee/index.php"> <img src="https://yt3.googleusercontent.com/ytc/AIdro_mlTtuvk8IoEtfAKvw7WDc0XuxqGBCjMn-klO63nQbfVjE=s176-c-k-c0x00ffffff-no-rj" width="100px;" style="border-radius: 50%;" alt=""/> <br /> _{<b>Hung-yi Lee</b>} </a> </td> </tr> </table>

:trident: 2026 News

New survey repo for spoken dialogue models is available on GitHub

<p align="center"> <img src="https://github.com/ga642381/Spoken-Dialogue-Model-Survey/blob/main/assets/sdm-timeline.png" width="500"> </p>

@article{chang2026tico,
      title={TiCo: Time-Controllable Training for Spoken Dialogue Models},
      author={Kai-Wei Chang and Wei-Chih Chen and En-Pei Hu and Hung-yi Lee and James Glass},
      journal={arXiv preprint arXiv:2603.22267},
      year={2026}
}

:trident: 2025 News

The survey paper “On The Landscape of Spoken Language Models: A Comprehensive Survey” is now available on arXiv

@article{arora2025landscape,
  title={On The Landscape of Spoken Language Models: A Comprehensive Survey},
  author={Arora, Siddhant and Chang, Kai-Wei and Chien, Chung-Ming and Peng, Yifan and Wu, Haibin and Adi, Yossi and Dupoux, Emmanuel and Lee, Hung-Yi and Livescu, Karen and Watanabe, Shinji},
  journal={arXiv preprint arXiv:2504.08528},
  year={2025}
}

It provides a comprehensive survey of spoken language models (SLMs), which covers a lot of speech/audio langauge models surveyed in this Speech-Trident project. But with more detailed and technical discussion. The paper categorizes the SLMs into:

1.⚡ Pure Speech LM

2.⚡ Speech-aware Text LM

3.⚡ Speech + Text LM

In addition, the paper discusses training strategies, speech/text token decoding patterns, duplex speech dialogue, benchmarks for SLMs, and more. Please read the paper for more details. You will enjoy it!!

:trident: Speech/Audio Language Models

| Date | Model Name | Paper Title | Link | | ------- | -------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | | 2025-07 | Audio Flamingo 3 | Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models | paper | | 2025-07 | DeSTA2.5-Audio | DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment | paper | | 2025-05 | SLED | Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space | paper | | 2025-05 | BALSa | From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data | paper | | 2025-05 | Fun-ASR-Nano | Fun-ASR-Nano: An LLM-based Non-autoregressive End-to-End Speech Recognition Model | paper, code | | 2025-04 | - | On The Landscape of Spoken Language Models: A Comprehensive Survey | Paper | | 2025-04 | Kimi-Audio | Kimi-Audio Technical Report | Paper | | 2025-03 | Qwen2.5-Omni | Qwen2.5-Omni Technical Report | Paper | | 2025-03 | Yue | YuE: Scaling Open Foundation Models for Long-Form Music Generation | Paper | | 2025-03 | CSM | Conversational speech generation | blog | | 2025-03 | Phi-4-Multimodal | Phi-4-Mini Technical Report: compact yet powerful multimodal language models via mixture-of-LoRAs | paper | | 2025-03 | Baichuan-Audio | Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction | paper | | 2025-02 | DiTAR | DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation | paper | | 2025-02 | Slamming | Slamming: Training a Speech Language Model on One GPU in a Day | paper | | 2025-02 | Step-Audio | Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction | paper | | 2025-01 | BAICHUAN-OMNI-1.5 | BAICHUAN-OMNI-1.5 TECHNICAL REPORT | paper | | 2025-01 | MiniCPM-o | A GPT-4o Level MLLM for Vision, Speech and Multimodal Live Streaming on Your Phone | GitHub | | 2025-01 | MinMo | MinMo: A Multimodal large Language Model for Seamless Voice Interaction | Paper | | 2025-01 | VITA-1.5 | VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction | Paper | | 2025-01 | OMNICHAT | OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios | Paper | | 2025-01 | SLIDE | SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation | Paper | | 2024-12 | SLAM-Omni | SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training | paper, code | | 2024-12 | TouchTTS | TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch | Paper | | 2024-12 | CosyVoice 2 | CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models | Paper | | 2024-12 | GLM-4-Voice | GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot | Paper | | 2024-12 | AlignFormer | AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM | Paper | | 2024-11 | -- | Scaling Speech-Text Pre-training with Synthetic Interleaved Data | Paper | | 2024-11 | -- | State-Space Large Audio Language Models | Paper | | 2024-11 | -- | Building a Taiwanese Mandarin Spoken Language Model: A First Attempt | Paper | | 2024-11 | Ultravox | Ultravox: An open-weight alternative to GPT-4o Realtime | Blog | | 2024-11 | hertz-dev | blog | GitHub | | 2024-11 | Freeze-Omni | Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM | paper | | 2024-11 | Align-SLM | Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback | [paper](https://arxiv.org/pdf/2