Speech to Text 🚀

This repo create ASR system by using toolkits PLUS own implementations.

Toolkits used:

Mozilla Deep Speech
DeepSpeech2
Wave2vec
Our Implementations

TIP : If you want to just use the scripts( text_cleaning / data_collecting/ create final dataset csv files) simply use requirements.txt to install important dependencies.

pip3 install requirements.txt

Prerequisites 📋

You need to know about RNNs, Attention mechanism, CTC, a little pandas/numpy, Tensorflow, KERAS and NLP Stuff(e.g. transformers, text cleaning etc. ). Also knowing about Spectrograms, MFCC, Filter Bank will help you to understand preprocess of audios. if you don't know any of these stuff you can check Wiki page there are many links

Models Output 🎯

<table width="100%"> <tr> <th width="30%"> model Name </th> <th width="40%"> DataSet </th> <th width="70%"> Loss </th> </tr> <tr> <td width="25%"> our own implementations : (first try) </td> <td width="25%"> common_voice_en (400h) </td> <td width="25%"> 74 ( So bad results / many models tested. ) </td> </tr> <tr> <td width="25%">Deep Speech 1 : Mozilla (first try) </td> <td width="25%"> common_voice + tv programms + radio programs (totally 300h) </td> <td width="25%"> 28 </td> </tr> <tr> <td width="25%">Deep Speech 1 : Mozilla (second try) </td> <td width="25%"> common_voice + tv programms + radio programs (totally 300h) </td> <td width="25%"> 25</td> </tr> <tr> <td width="25%">Deep Speech 1 : Mozilla + Transfer Learning (third try) </td> <td width="25%"> common_voice + tv programms + radio programs (totally 300h) </td> <td width="25%"> 24</td> </tr> <tr> <td width="25%">Deep Speech 1 : Mozilla + Transfer Learning (third try) </td> <td width="25%"> common_voice + tv programms + radio programs (totally +1000h) </td> <td width="25%"> 22 </td> </tr> </table>

Dataset we used 📁

there are many public datasets for English. But for persian there is not enough and free STT dataset.So we created our own data crawler for collecting data.

common voice dataset is a rich free dataset.

How to use our script to collect data

in this repo there is 1 folders:

crawler

crawler : this folder has one script. the script will crawl in radio archive and collect the data we need. you can edit this crawler to download other websites too. [More info Check README file in crawler folder]

Full DataSet 📁 ⚡🔥

Here in Hamtech Company, we decided to open source our ASR dataset. This Dataset is near 200Gb of voice plus CSV file which includes the transcription(some files contain txt file not csv). you can find a column named Confidence_level, this means how much the transcription is reliable, here is the, you can use LM(language models) or any other idea to clean them or any other ideas. . In conclusion :

You may use some techniques ( like using LMs, using annotator, etc. ) to clean more the transcription
you can use the confidence_level column in the CSV file to select more accurate rows.
wav files are nearly 200Gb.
voices format is : format : Wav / channels : mono/ sample_rate : 16000 Hz/

Note : 9 Gb of data is lost. :(

Links : Version 1 data content is like : WavFile+TxtFile | Version 2 data content is like : ZipFile+CsvFile

Dataset_part_1_v1 : https://drive.google.com/drive/folders/1jdR4joj1BsU_LYHXriUW0xT4tgXicaqv?usp=share_link
Dataset_part_2_v1 : https://drive.google.com/drive/folders/1tVOrcwpxVcfrON5t9rdFSLZ5LNwJuCrV?usp=share_link
Dataset_part_3_v1 : https://drive.google.com/drive/folders/1FWY3MTrpMF-WrqFbMSM-fNmLn0FiUoNx?usp=share_link
Dataset_part_4_v1 : Data is lost :( ( this part of data was 9.7 Gb of zipped wav/txt files )
Dataset_part_1_v2 : https://drive.google.com/drive/folders/1ZsTMb_V-UAXxxi-wRE-g4hXXntonA_P3?usp=share_link
Dataset_part_2_v2 : https://drive.google.com/drive/folders/1eAPjF_DVU9j4nQ8S0aWQTbCbTI5sBrYp?usp=share_link
Dataset_part_3_v2 : https://drive.google.com/drive/folders/1rMNYwKtkyz8tprhwErrcDT-TLKtWA0OB?usp=share_link
Dataset_part_4_v2 : https://drive.google.com/drive/folders/1Lxq8ouA6UWEOkHfNjxJ7Kf5k51D5t2V8?usp=share_link

NOTE : if you need more tips dont hesitate to email me : masoudparpanchi@gmail.com

Part of Our Dataset V0.1 📁 ⚡🔥

Here in Hamtech Company, we decided to open source a challenging part of our ASR dataset. This Dataset is near 30 Hours of voice plus CSV file which includes the transcription. you can find a column named Confidence_level, this means how much the transcription is reliable, here is the, you can use LM(language models) or any other idea to clean them or any other ideas. The variety of speakers in this dataset is not so much But the quality of voices is good enough. Check Dataset Folder in this repo. In conclusion :

Dataset is near 30H
Transcriptions are not an exact match.
Variety of speakers in this dataset is not so much
You may use some techniques ( like using LMs, using annotator, etc. ) to clean more the transcription
you can use the confidence_level column in the CSV file to select more accurate rows.
wav files are nearly 2Gb.
voices format is : format : Wav / channels : mono/ sample_rate : 16000 Hz/
Google drive dataset URL is https://drive.google.com/drive/folders/1BLOYLBOUSWI50k4RTnpTc7Ni4rYxjVi2?usp=sharing
to download just the CSV file check Dataset folder in this repo or the google drive link

Mozilla Deep Speech

Last checkpoint of trained Speech to text ( These are not ready to use for commercial usecases. only a finetuned model for you, to use it in your own project :

Mozzila deepspeech checkpoints ( WARNING : Check experiments log for hyperparametrs when you want to finetune using these checkpoints ) : https://drive.google.com/drive/folders/1FyLFudV_o71WeBQEQIn_-GficBpEMIzM?usp=share_link
pb format of checkpoint : https://drive.google.com/file/d/1RhISAEUwG9MwkLIFyrb1sIFi4UhNTKL6/view?usp=share_link

Start Using deepspeech : clone and download common voice dataset

To use this toolkit you must first do what this link says or follow short Installation blow.

Currently we are using DeepSpeech V0.9.3

Short Installation

Clone and create virtual enviroments

git clone --branch v0.9.3 https://github.com/mozilla/DeepSpeech
cd DeepSpeech
pip3 install --upgrade pip==20.2.2 wheel==0.34.2 setuptools==49.6.0
pip3 install --upgrade -e .
pip install tensorflow-gpu==1.15.4

After cloning and install dependecies you need common voice dataset. Download the proper language and then preprocessing. all the steps for preprocess common voice dataset is documented in here too.

Start training from scratch ( no pretrained model)

‫‪python3‬‬ ‫‪DeepSpeech.py‬‬ ‫‪--train_files‬‬ ‫‪../data/CV/en/clips/train.csv‬‬ ‫‪--dev_files‬‬ \
‫‪../data/CV/en/clips/dev.csv‬‬ ‫‪--test_files‬‬ ‫‪../data/CV/en/clips/test.csv‬‬

your own dataset

if you want to create your own dataset you need these TIPS :

all your audio files must be in wav format
your audios must be in : 16khz / 16bitpersample / mono / no noise ( optional )
don't use more than 30 minutes of audio files recorded by one person
at least you must have 300h of data.
you need a csv file for all the audio files you have. you can see the structure of csv file after you preprocessed the common voice files by import_cv2.py( this python file is in Deepspeech repo. you'll have it after cloning)
and also you need to clean your transcripts (Some more NLP task :) ).
(optional) if you want to use my crawler to create dataset you can check crawler.py (in crawler folder ) and parallel.py(in data_collector folder) file.
(optional) if you want to use my transcription desktop application you can check ciacada repository
( optional ) you can use my scripts to clean transcripts and make final csv file. they are available in github
don't worry about creating spectrogram/FastFourier/MFCC/Filterbank , Mozilla will do it all for you.
TODO : need to remember more tips.

Language model

you need language model for testing the model. the language model is trained on Kenlm

the steps to train language model is here

text file size : 2GB

test file to optimize : 20M

kenlm checkpoint link (This checkpoint is just a toy language model. trained on 2.5 Gb of Persian txt but not deep optimization ): https://drive.google.com/file/d/1IGL_SXNQdYINWEP93JnbAw1NjxtmZ-Hw/view?usp=share_link

txt dataset to train kenlm can be found here ( you can find near 80 Gb of Persian txt there ): https://nlpdataset.ir/farsi/raw_text_corpora.html

after all these steps it means you have your dataset ready and you want to train . if you want to train on English there is no more steps BUT if you are in another language ( like Persian ) you need to check transfer learning part of this link TIP : don't forget to change alphabete.txt

Question : can I use Other languages checkpoints to start transfer learning? Sure, do it. But remember to drop weights of last N layers.

you may need the meaning of flags to use all the abilities of mozilla deep speech. check their Documentation

Tip : if you faced with CUDA/CudNN errors. try to use conda and install proper versions.

Speech2text

Install / Use

README