Simultaneous Speech Translation

Code base for simultaneous speech translation experiments. It is based on fairseq.

Implemented

Encoder

• Transformers with convolutional context for ASR
• Emformer

Streaming Models

• Wait-k [example]
• Monotonic Multihead Attention [example]
• Continuous Integrate-and-Fire [example]

Setup

1. Install fairseq

git clone https://github.com/pytorch/fairseq.git
cd fairseq
git checkout 4a7835b
python setup.py build_ext --inplace
pip install .

2. (Optional) Install apex for faster mixed precision (fp16) training.
3. Install dependencies

pip install -r requirements.txt

4. Update submodules

git submodule update --init --recursive

Pre-trained model

ASR model with Emformer encoder and Transformer decoder. Pre-trained with joint CTC cross-entropy loss. |MuST-C (WER)|en-de (V2)|en-es| |-|-|-| |dev|9.65|14.44| |tst-COMMON|12.85|14.02| |model|download|download| |vocab|download|download|

Sequence-level Knowledge Distillation

|MuST-C (BLEU)|en-de (V2)| |-|-| |valid|31.76| |distillation|download| |vocab|download|

Citation

Please consider citing our paper:

@inproceedings{chang22f_interspeech,
  author={Chih-Chiang Chang and Hung-yi Lee},
  title={{Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={5175--5179},
  doi={10.21437/Interspeech.2022-10627}
}