SpeechColab ASR leaderboard

1. Overview

"If you can’t measure it, you can’t improve it." -- Peter Drucker

SpeechIO leaderboard serves as an ASR benchmarking platform by providing 3 components:

1. TestSet Zoo: A collection of test sets covering wide range of speech recognition tasks & scenarios

2. Model Zoo: A collection of models including commercial APIs & open-sourced models

3. Benchmarking Pipeline: a simple & well-specified pipeline to take care of data preparation / recognition / post processing / error rate evaluation.

People should be able to easily benchmark, reproduce, examine ASR systems from each other

---

2. TestSet Zoo: datasets/*

<details><summary> Academic Test Sets (EN & ZH) </summary><p>

| 已公开 <br> UNLOCKED | 编号 <br> DATASET_ID | 说明 <br> DESCRIPTION | 语言 <br> LANGUAGE | | --- | --- | --- | --- | | ✓ | AISHELL1_TEST | test set of AISHELL-1 | zh | | ✓ | AISHELL2_IOS_TEST | test set of AISHELL-2 (iOS channel) | zh | | ✓ | AISHELL2_ANDROID_TEST | test set of AISHELL-2 (Android channel) | zh | | ✓ | AISHELL2_MIC_TEST | test set of AISHELL-2 (Microphone channel) | zh | | ✓ | ALIMEETING_EVAL_NEAR_FIELD | AliMeeting | zh | | ✓ | ALIMEETING_TEST_NEAR_FIELD | AliMeeting | zh | | ✓ | ALIMEETING_EVAL_FAR_FIELD | AliMeeting | zh | | ✓ | ALIMEETING_TEST_FAR_FIELD | AliMeeting | zh | | ✓ | LIBRISPEECH_TEST_CLEAN | "test_clean" set of LibriSpeech | en | | ✓ | LIBRISPEECH_TEST_OTHER | "test_other" set of LibriSpeech | en | | ✓ | TEDLIUM_RELEASE3_LEGACY_DEV | tedlium release 3, legacy dir dev set TEDLium3 | en | | ✓ | TEDLIUM_RELEASE3_LEGACY_TEST | tedlium release 3, legacy dir test set TEDLium3 | en | | ✓ | GIGASPEECH_V1.0.0_DEV | dev set of GigaSpeech | en | | ✓ | GIGASPEECH_V1.0.0_TEST | test set of GigaSpeech | en | | ✓ | VOXPOPULI_V1.0_EN_DEV | dev set of VoxPopuli | en | | ✓ | VOXPOPULI_V1.0_EN_TEST | test set of VoxPopuli | en | | ✓ | VOXPOPULI_V1.0_EN_ACCENTED_TEST | accented test set of VoxPopuli | en | | ✓ | COMMON_VOICE_V11.0_DEV | dev set of Common Voice | en | | ✓ | COMMON_VOICE_V11.0_TEST | test set of Common Voice | en |

</p></details> <details><summary> SpeechIO Test Sets (ZH) </summary><p>

SpeechIO test sets are carefully curated by SpeechIO authors, crawled from publicly available sources (Youtube, TV programs, Podcast etc), covering various well-known scenarios and topics, transcribed by payed professional annotators.

| 已公开 <br> UNLOCKED | 编号 <br> DATASET_ID | 名称 <br> NAME | 场景 <br> SCENARIO | 内容领域 <br> TOPIC | 有效时长 <br> DURATION (HOURS) | 难度(1-5) <br> DIFFICULTY | | --- | --- | --- | --- | --- | --- | --- | | ✓ |SPEECHIO_ASR_ZH00000| 调试集 <br> for debugging | 视频会议、论坛演讲 <br> conference & speech | 经济、货币、金融 <br> economy, currency, finance | 1.0 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00001| 新闻联播 | 新闻播报 <br> TV News | 时政 <br> news & politics | 9 | ★ | | ✓ |SPEECHIO_ASR_ZH00002| 鲁豫有约 | 访谈电视节目 <br> TV interview | 名人工作/生活 <br> celebrity & film & music & daily | 3 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00003| 天下足球 | 专题电视节目 <br> TV program | 足球 <br> Sports & Football & Worldcup | 2.7 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00004| 罗振宇跨年演讲 | 会场演讲 <br> Stadium Public Speech | 社会、人文、商业 <br> Society & Culture & Business Trend | 2.7 | ★★ | | ✓ |SPEECHIO_ASR_ZH00005| 李永乐讲堂 | 在线教育 <br> Online Education | 科普 <br> Popular Science | 4.4 | ★★★ | | ✓ |SPEECHIO_ASR_ZH00006| 王者荣耀 <br> 张大仙 & 骚白 | 直播 <br> Live Broadcasting | 游戏 <br> Game | 1.6 | ★★★☆ | | ✓ |SPEECHIO_ASR_ZH00007| 直播带货 <br> 李佳琪 & 薇娅 | 直播 <br> Live Broadcasting | 电商、美妆 <br> Makeup & Online shopping/advertising | 0.9 | ★★★★☆ | | ✓ |SPEECHIO_ASR_ZH00008| 老罗语录 | 线下培训 <br> Offline lecture | 段子、做人 <br> Life & Purpose & Ethics | 1.3 | ★★★★☆ | | ✓ |SPEECHIO_ASR_ZH00009| 故事FM | 播客 <br> Podcast | 人生故事、见闻 <br> Ordinary Life Story Telling | 4.5 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00010| 创业内幕 | 播客 <br> Podcast | 创业、产品、投资 <br> Startup & Enterprenuer & Product & Investment | 4.2 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00011| 罗翔刑法法考 | 在线教育 <br> Online Education | 法律 法考 <br> Law & Lawyer Qualification Exams | 3.4 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00012| 张雪峰考研 | 在线教育 <br> Online Education | 考研 高校报考 <br> University & Graduate School Entrance Exams | 3.4 | ★★★☆ | | ✓ |SPEECHIO_ASR_ZH00013| 谷阿莫 <br> 牛叔说电影 | 短视频 <br> VLog | 电影剪辑 <br> Movie Cuts | 1.8 | ★★★ | | ✓ |SPEECHIO_ASR_ZH00014| 贫穷料理 <br> 琼斯爱生活 | 短视频 <br> VLog | 美食、烹饪 <br> Food & Cooking & Gourmet | 1 | ★★★☆ | | ✓ |SPEECHIO_ASR_ZH00015| 单田芳 白眉大侠 | 评书 <br> Traditional Podcast | 江湖、武侠 <br> Kongfu Fiction | 2.2 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00016| 德云社演出 | 剧场相声 <br> Theater Crosstalk Show | 包袱段子 <br> Funny Stories | 1 | ★★★ | | ✓ |SPEECHIO_ASR_ZH00017| 吐槽大会 | 脱口秀电视节目 <br> Standup Comedy | 明星糗事 <br> Celebrity Jokes | 1.8 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00018| 小猪佩奇 <br> 熊出没 | 少儿动画 <br> Children Cartoon | 童话故事、日常 <br> Fairy Tale | 0.9 | ★☆ | | ✓ |SPEECHIO_ASR_ZH00019| CCTV5 NBA 转播 | 体育赛事解说 <br> Sports Game Live | 篮球、NBA <br> NBA Game | 0.7 | ★★★ | | ✓ |SPEECHIO_ASR_ZH00020| 篮球人物 | 纪录片 <br> Documentary | 篮球明星、成长 <br> NBA Super Stars' Life & History | 2.2 | ★★ | | ✓ |SPEECHIO_ASR_ZH00021| 汽车之家评测 | 短视频 <br> VLog | 汽车测评 <br> Car benchmarks, Road driving test | 1.7 | ★★★☆ | | ✓ |SPEECHIO_ASR_ZH00022| 小艾大叔 豪宅带看 | 短视频 <br> VLog | 房地产、豪宅 <br> Realestate, Mansion tour | 1.7 | ★★★ | | ✓ |SPEECHIO_ASR_ZH00023| 无聊开箱 <br> Zealer评测 | 短视频 <br> VLog | 产品开箱评测 <br> Unboxing | 2 | ★★★ | | ✓ |SPEECHIO_ASR_ZH00024| 付老师种植技术 | 短视频 <br> VLog | 农业、种植 <br> Agriculture, Planting | 2.7 | ★★★☆ | | ✓ |SPEECHIO_ASR_ZH00025| 石国鹏讲历史 | 线下培训 <br> Offline lecture | 历史，古希腊哲学 <br> History, Greek philosophy | 1.3 | ★★☆ | | ✓ |SPEECHIO_ASR_ZH00026| 张震鬼故事 | 广播节目 <br> Broadcasting Program | 鬼故事 <br> Horror Stories | 2.4 | ★★★ | | ✗ |SPEECHIO_ASR_ZH00027| 华语辩论世界杯 | 辩论赛 <br> Debates Contest | 兴趣、技能、成长 <br> Hobby, Skill, Growth | 1.4 | ★★★ | | ✗ |SPEECHIO_ASR_ZH00028| 时政现场同传 | 同声传译 <br> Simultaneous Translation | 时政、社会公共治理 <br> News & Events on Public Governance | 2.1 | ★★★☆ | | ✗ |SPEECHIO_ASR_ZH00029| 港台明星访谈 <br> 周杰伦,曾志伟 <br> 张家辉,陈小春 <br> 周星驰 | 口音(港台) <br> HongKong/Taiwan Accents | 娱乐、生活、演艺 <br> Entertainment, Acting, Musics | 1.5 | ★★★☆ | | ✗ |SPEECHIO_ASR_ZH00030| 世界青年说 | 口音(老外) <br> Foreigner Accents | 异国文化比较 <br> Cultural Difference | 2 | ★★★☆ | | ✗ |SPEECHIO_ASR_ZH00031| 东方甄选 | 直播 <br> broadcast | 带货，英语教学 <br> Online advertising & English Education | 2.4 | ★★★☆ | | ✗ |SPEECHIO_ASR_ZH00032| 郎朗钢琴课 | 长视频 <br> long-form video | 音乐乐理，钢琴 <br> Music & piano | 1.7 | ★★☆ | | ✗ |SPEECHIO_ASR_ZH00033| 老石谈芯 | 短视频 <br> VLog | 芯片 <br> chips | 2.8 | ★★★ | | ✗ |SPEECHIO_ASR_ZH00034| 电丸科技AK | 短视频 <br> VLog | 网络 IT <br> Internet tech, IT | 1.4 | ★★★☆ | | ✗ |SPEECHIO_ASR_ZH00035| 新氧医美 | 短视频 <br> VLog | 医疗美容 <br> Medical Cosmetology | 1.4 | ★★ | | ✗ |SPEECHIO_ASR_ZH00036| 交通广播 | 交通广播 <br> traffic radio | 路况，娱乐 <br> Traffics | 1.2 | ★★★☆ | | ✗ |SPEECHIO_ASR_ZH00037| 老俞闲聊 | 在线会议 <br> Online meeting | 闲聊 <br> chat | 2.4 | ★★★ | | ✗ |SPEECHIO_ASR_ZH00038| 电影：疯狂石头+疯狂赛车 | 电影 <br> Film | 重庆话、山东青岛、四川成都话、河北唐山话、粤语、天津话、河南话、陕西话、闽南话，武汉话等 <br> multiple accents | 1.3 | ★★★★☆ | | ✗ |SPEECHIO_ASR_ZH00039| 电影：1942 | 电影 <br> Film | 河南话 <br> HeNan Accent | 0.9 | ★★★★ | | ✗ |SPEECHIO_ASR_ZH00040| 电影：白鹿原 | 电影 <br> Film | 陕西话 <br> ShaanXi Accent | 1.1 | ★★★★★ | | ✗ |SPEECHIO_ASR_ZH00041| 电影：让子弹飞 | 电影 <br> Film | 四川话 <br> SiChuan Accent | 1.1 | ★★★★☆ | | ✗ |SPEECHIO_ASR_ZH00042| 电影：人生大事 | 电影 <br> Film | 武汉话 <br> WuHan Accent | 0.8 | ★★★★ | | ✗ |SPEECHIO_ASR_ZH00043| 听障 | 听障语音识别 <br> Hearing Imperiment Speaker | 新闻脚本 <br> News Prompts | 0.6 | ★★★★★ | | ✗ |SPEECHIO_ASR_ZH00044| 唐诗宋词 | 诗词朗诵 <br> Poems Reading | 唐诗宋词 <br> Chinese Poems | 1.1 | ★★★☆ | | ✗ |SPEECHIO_ASR_ZH00045| 文言文 | 文言文朗诵 <br> Classical Chinese Reading | 论语，老子，诗经，孙子兵法 | 0.5 | ★★★★★ | | ✗ |SPEECHIO_ASR_ZH00046| 音乐歌词识别 | 演唱 <br> Singing | 歌词 <br> Lyrics | 1.2 | ★★★★☆ |

</p></details>

---

3. Model Zoo: models/*

<details><summary> EN Models </summary><p>

| 编号 <br> MODEL_ID | 类型 <br> TYPE | 厂商/作者 <br> PROVIDER/AUTHOR | 简介 <br> DESCRIPTION | 链接 <br> URL | | --- | --- | --- | --- | --- | | aliyun_api_en | Cloud | Alibaba | | link | | amazon_api_en | Cloud | Amazon AWS | | link | | baidu_api_en | Cloud | Baidu | | link | | google_api_en | Cloud | Google | | link | | google_USM_en | Cloud | Google | | request access | | microsoft_sdk_en | Cloud | Microsoft Azure | | link | | tencent_api_en | Cloud | Tencent | | [link](https://cloud.tence