Upload tools/asr\models\speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch\README.md with huggingface_hub
Browse files
tools/asr//models//speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch//README.md
ADDED
|
@@ -0,0 +1,407 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
tasks:
|
| 3 |
+
- auto-speech-recognition
|
| 4 |
+
domain:
|
| 5 |
+
- audio
|
| 6 |
+
model-type:
|
| 7 |
+
- Non-autoregressive
|
| 8 |
+
frameworks:
|
| 9 |
+
- pytorch
|
| 10 |
+
backbone:
|
| 11 |
+
- transformer/conformer
|
| 12 |
+
metrics:
|
| 13 |
+
- CER
|
| 14 |
+
license: Apache License 2.0
|
| 15 |
+
language:
|
| 16 |
+
- cn
|
| 17 |
+
tags:
|
| 18 |
+
- FunASR
|
| 19 |
+
- Paraformer
|
| 20 |
+
- Alibaba
|
| 21 |
+
- INTERSPEECH 2022
|
| 22 |
+
datasets:
|
| 23 |
+
train:
|
| 24 |
+
- 60,000 hour industrial Mandarin task
|
| 25 |
+
test:
|
| 26 |
+
- AISHELL-1 dev/test
|
| 27 |
+
- AISHELL-2 dev_android/dev_ios/dev_mic/test_android/test_ios/test_mic
|
| 28 |
+
- WentSpeech dev/test_meeting/test_net
|
| 29 |
+
- SpeechIO TIOBE
|
| 30 |
+
- 60,000 hour industrial Mandarin task
|
| 31 |
+
indexing:
|
| 32 |
+
results:
|
| 33 |
+
- task:
|
| 34 |
+
name: Automatic Speech Recognition
|
| 35 |
+
dataset:
|
| 36 |
+
name: 60,000 hour industrial Mandarin task
|
| 37 |
+
type: audio # optional
|
| 38 |
+
args: 16k sampling rate, 8404 characters # optional
|
| 39 |
+
metrics:
|
| 40 |
+
- type: CER
|
| 41 |
+
value: 8.53% # float
|
| 42 |
+
description: greedy search, withou lm, avg.
|
| 43 |
+
args: default
|
| 44 |
+
- type: RTF
|
| 45 |
+
value: 0.0251 # float
|
| 46 |
+
description: GPU inference on V100
|
| 47 |
+
args: batch_size=1
|
| 48 |
+
widgets:
|
| 49 |
+
- task: auto-speech-recognition
|
| 50 |
+
inputs:
|
| 51 |
+
- type: audio
|
| 52 |
+
name: input
|
| 53 |
+
title: 音频
|
| 54 |
+
examples:
|
| 55 |
+
- name: 1
|
| 56 |
+
title: 示例1
|
| 57 |
+
inputs:
|
| 58 |
+
- name: input
|
| 59 |
+
data: https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav
|
| 60 |
+
inferencespec:
|
| 61 |
+
cpu: 8 #CPU数量
|
| 62 |
+
memory: 4096
|
| 63 |
+
finetune-support: True
|
| 64 |
+
---
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
# Paraformer-large模型介绍
|
| 68 |
+
|
| 69 |
+
## Highlights
|
| 70 |
+
- 热词版本:[Paraformer-large热词版模型](https://www.modelscope.cn/models/damo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/summary)支持热词定制功能,基于提供的热词列表进行激励增强,提升热词的召回率和准确率。
|
| 71 |
+
- 长音频版本:[Paraformer-large长音频模型](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary),集成VAD、ASR、标点与时间戳功能,可直接对时长为数小时音频进行识别,并输出带标点文字与时间戳。
|
| 72 |
+
|
| 73 |
+
## <strong>[FunASR开源项目介绍](https://github.com/alibaba-damo-academy/FunASR)</strong>
|
| 74 |
+
<strong>[FunASR](https://github.com/alibaba-damo-academy/FunASR)</strong>希望在语音识别的学术研究和工业应用之间架起一座桥梁。通过发布工业级语音识别模型的训练和微调,研究人员和开发人员可以更方便地进行语音识别模型的研究和生产,并推动语音识别生态的发展。让语音识别更有趣!
|
| 75 |
+
|
| 76 |
+
[**github仓库**](https://github.com/alibaba-damo-academy/FunASR)
|
| 77 |
+
| [**最新动态**](https://github.com/alibaba-damo-academy/FunASR#whats-new)
|
| 78 |
+
| [**环境安装**](https://github.com/alibaba-damo-academy/FunASR#installation)
|
| 79 |
+
| [**服务部署**](https://www.funasr.com)
|
| 80 |
+
| [**模型库**](https://github.com/alibaba-damo-academy/FunASR/tree/main/model_zoo)
|
| 81 |
+
| [**联系我们**](https://github.com/alibaba-damo-academy/FunASR#contact)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
## 模型原理介绍
|
| 85 |
+
|
| 86 |
+
Paraformer是达摩院语音团队提出的一种高效的非自回归端到端语音识别框架。本项目为Paraformer中文通用语音识别模型,采用工业级数万小时的标注音频进行模型训练,保证了模型的通用识别效果。模型可以被应用于语音输入法、语音导航、智能会议纪要等场景。
|
| 87 |
+
|
| 88 |
+
<p align="center">
|
| 89 |
+
<img src="fig/struct.png" alt="Paraformer模型结构" width="500" />
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
Paraformer模型结构如上图所示,由 Encoder、Predictor、Sampler、Decoder 与 Loss function 五部分组成。Encoder可以采用不同的网络结构,例如self-attention,conformer,SAN-M等。Predictor 为两层FFN,预测目标文字个数以及抽取目标文字对应的声学向量。Sampler 为无可学习参数模块,依据输入的声学向量和目标向量,生产含有语义的特征向量。Decoder 结构与自回归模型类似,为双向建模(自回归为单向建模)。Loss function 部分,除了交叉熵(CE)与 MWER 区分性优化目标,还包括了 Predictor 优化目标 MAE。
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
其核心点主要有:
|
| 96 |
+
- Predictor 模块:基于 Continuous integrate-and-fire (CIF) 的 预测器 (Predictor) 来抽取目标文字对应的声学特征向量,可以更加准确的预测语音中目标文字个数。
|
| 97 |
+
- Sampler:通过采样,将声学特征向量与目标文字向量变换成含有语义信息的特征向量,配合双向的 Decoder 来增强模型对于上下文的建模能力。
|
| 98 |
+
- 基于负样本采样的 MWER 训练准则。
|
| 99 |
+
|
| 100 |
+
更详细的细节见:
|
| 101 |
+
- 论文: [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://arxiv.org/abs/2206.08317)
|
| 102 |
+
- 论文解读:[Paraformer: 高识别率、高计算效率的单轮非自回归端到端语音识别模型](https://mp.weixin.qq.com/s/xQ87isj5_wxWiQs4qUXtVw)
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
## 基于ModelScope进行推理
|
| 106 |
+
|
| 107 |
+
- 推理支持音频格式如下:
|
| 108 |
+
- wav文件路径,例如:data/test/audios/asr_example.wav
|
| 109 |
+
- pcm文件路径,例如:data/test/audios/asr_example.pcm
|
| 110 |
+
- wav文件url,例如:https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav
|
| 111 |
+
- wav二进制数据,格式bytes,例如:用户直接从文件里读出bytes数据或者是麦克风录出bytes数据。
|
| 112 |
+
- 已解析的audio音频,例如:audio, rate = soundfile.read("asr_example_zh.wav"),类型为numpy.ndarray或者torch.Tensor。
|
| 113 |
+
- wav.scp文件,需符合如下要求:
|
| 114 |
+
|
| 115 |
+
```sh
|
| 116 |
+
cat wav.scp
|
| 117 |
+
asr_example1 data/test/audios/asr_example1.wav
|
| 118 |
+
asr_example2 data/test/audios/asr_example2.wav
|
| 119 |
+
...
|
| 120 |
+
```
|
| 121 |
+
|
| 122 |
+
- 若输入格式wav文件url,api调用方式可参考如下范例:
|
| 123 |
+
|
| 124 |
+
```python
|
| 125 |
+
from modelscope.pipelines import pipeline
|
| 126 |
+
from modelscope.utils.constant import Tasks
|
| 127 |
+
|
| 128 |
+
inference_pipeline = pipeline(
|
| 129 |
+
task=Tasks.auto_speech_recognition,
|
| 130 |
+
model='iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch', model_revision="v2.0.4")
|
| 131 |
+
|
| 132 |
+
rec_result = inference_pipeline('https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav')
|
| 133 |
+
print(rec_result)
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
- 输入音频为pcm格式,调用api时需要传入音频采样率参数audio_fs,例如:
|
| 137 |
+
|
| 138 |
+
```python
|
| 139 |
+
rec_result = inference_pipeline('https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.pcm', audio_fs=16000)
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
- 输入音频为wav格式,api调用方式可参考如下范例:
|
| 143 |
+
|
| 144 |
+
```python
|
| 145 |
+
rec_result = inference_pipeline('asr_example_zh.wav')
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
- 若输入格式为文件wav.scp(注:文件名需要以.scp结尾),可添加 output_dir 参数将识别结果写入文件中,api调用方式可参考如下范例:
|
| 149 |
+
|
| 150 |
+
```python
|
| 151 |
+
inference_pipeline("wav.scp", output_dir='./output_dir')
|
| 152 |
+
```
|
| 153 |
+
识别结果输出路径结构如下:
|
| 154 |
+
|
| 155 |
+
```sh
|
| 156 |
+
tree output_dir/
|
| 157 |
+
output_dir/
|
| 158 |
+
└── 1best_recog
|
| 159 |
+
├── score
|
| 160 |
+
└── text
|
| 161 |
+
|
| 162 |
+
1 directory, 3 files
|
| 163 |
+
```
|
| 164 |
+
score:识别路径得分
|
| 165 |
+
|
| 166 |
+
text:语音识别结果文件
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
- 若输入音频为已解析的audio音频,api调用方式可参考如下范例:
|
| 170 |
+
|
| 171 |
+
```python
|
| 172 |
+
import soundfile
|
| 173 |
+
|
| 174 |
+
waveform, sample_rate = soundfile.read("asr_example_zh.wav")
|
| 175 |
+
rec_result = inference_pipeline(waveform)
|
| 176 |
+
```
|
| 177 |
+
|
| 178 |
+
- ASR、VAD、PUNC模型自由组合
|
| 179 |
+
|
| 180 |
+
可根据使用需求对VAD和PUNC标点模型进行自由组合,使用方式如下:
|
| 181 |
+
```python
|
| 182 |
+
inference_pipeline = pipeline(
|
| 183 |
+
task=Tasks.auto_speech_recognition,
|
| 184 |
+
model='iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch', model_revision="v2.0.4",
|
| 185 |
+
vad_model='iic/speech_fsmn_vad_zh-cn-16k-common-pytorch', vad_model_revision="v2.0.4",
|
| 186 |
+
punc_model='iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch', punc_model_revision="v2.0.3",
|
| 187 |
+
# spk_model="iic/speech_campplus_sv_zh-cn_16k-common",
|
| 188 |
+
# spk_model_revision="v2.0.2",
|
| 189 |
+
)
|
| 190 |
+
```
|
| 191 |
+
若不使用PUNC模型,可配置punc_model="",或不传入punc_model参数,如需加入LM模型,可增加配置lm_model='damo/speech_transformer_lm_zh-cn-common-vocab8404-pytorch',并设置lm_weight和beam_size参数。
|
| 192 |
+
|
| 193 |
+
## 基于FunASR进行推理
|
| 194 |
+
|
| 195 |
+
下面为快速上手教程,测试音频([中文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav),[英文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav))
|
| 196 |
+
|
| 197 |
+
### 可执行命令行
|
| 198 |
+
在命令行终端执行:
|
| 199 |
+
|
| 200 |
+
```shell
|
| 201 |
+
funasr +model=paraformer-zh +vad_model="fsmn-vad" +punc_model="ct-punc" +input=vad_example.wav
|
| 202 |
+
```
|
| 203 |
+
|
| 204 |
+
注:支持单条音频文件识别,也支持文件列表,列表为kaldi风格wav.scp:`wav_id wav_path`
|
| 205 |
+
|
| 206 |
+
### python示例
|
| 207 |
+
#### 非实时语音识别
|
| 208 |
+
```python
|
| 209 |
+
from funasr import AutoModel
|
| 210 |
+
# paraformer-zh is a multi-functional asr model
|
| 211 |
+
# use vad, punc, spk or not as you need
|
| 212 |
+
model = AutoModel(model="paraformer-zh", model_revision="v2.0.4",
|
| 213 |
+
vad_model="fsmn-vad", vad_model_revision="v2.0.4",
|
| 214 |
+
punc_model="ct-punc-c", punc_model_revision="v2.0.4",
|
| 215 |
+
# spk_model="cam++", spk_model_revision="v2.0.2",
|
| 216 |
+
)
|
| 217 |
+
res = model.generate(input=f"{model.model_path}/example/asr_example.wav",
|
| 218 |
+
batch_size_s=300,
|
| 219 |
+
hotword='魔搭')
|
| 220 |
+
print(res)
|
| 221 |
+
```
|
| 222 |
+
注:`model_hub`:表示模型仓库,`ms`为选择modelscope下载,`hf`为选择huggingface下载。
|
| 223 |
+
|
| 224 |
+
#### 实时语音识别
|
| 225 |
+
|
| 226 |
+
```python
|
| 227 |
+
from funasr import AutoModel
|
| 228 |
+
|
| 229 |
+
chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
|
| 230 |
+
encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention
|
| 231 |
+
decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attention
|
| 232 |
+
|
| 233 |
+
model = AutoModel(model="paraformer-zh-streaming", model_revision="v2.0.4")
|
| 234 |
+
|
| 235 |
+
import soundfile
|
| 236 |
+
import os
|
| 237 |
+
|
| 238 |
+
wav_file = os.path.join(model.model_path, "example/asr_example.wav")
|
| 239 |
+
speech, sample_rate = soundfile.read(wav_file)
|
| 240 |
+
chunk_stride = chunk_size[1] * 960 # 600ms
|
| 241 |
+
|
| 242 |
+
cache = {}
|
| 243 |
+
total_chunk_num = int(len((speech)-1)/chunk_stride+1)
|
| 244 |
+
for i in range(total_chunk_num):
|
| 245 |
+
speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
|
| 246 |
+
is_final = i == total_chunk_num - 1
|
| 247 |
+
res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size, encoder_chunk_look_back=encoder_chunk_look_back, decoder_chunk_look_back=decoder_chunk_look_back)
|
| 248 |
+
print(res)
|
| 249 |
+
```
|
| 250 |
+
|
| 251 |
+
注:`chunk_size`为流式延时配置,`[0,10,5]`表示上屏实时出字粒度为`10*60=600ms`,未来信息为`5*60=300ms`。每次推理输入为`600ms`(采样点数为`16000*0.6=960`),输出为对应文字,最后一个语音片段输入需要设置`is_final=True`来强制输出最后一个字。
|
| 252 |
+
|
| 253 |
+
#### 语音端点检测(非实时)
|
| 254 |
+
```python
|
| 255 |
+
from funasr import AutoModel
|
| 256 |
+
|
| 257 |
+
model = AutoModel(model="fsmn-vad", model_revision="v2.0.4")
|
| 258 |
+
|
| 259 |
+
wav_file = f"{model.model_path}/example/asr_example.wav"
|
| 260 |
+
res = model.generate(input=wav_file)
|
| 261 |
+
print(res)
|
| 262 |
+
```
|
| 263 |
+
|
| 264 |
+
#### 语音端点检测(实时)
|
| 265 |
+
```python
|
| 266 |
+
from funasr import AutoModel
|
| 267 |
+
|
| 268 |
+
chunk_size = 200 # ms
|
| 269 |
+
model = AutoModel(model="fsmn-vad", model_revision="v2.0.4")
|
| 270 |
+
|
| 271 |
+
import soundfile
|
| 272 |
+
|
| 273 |
+
wav_file = f"{model.model_path}/example/vad_example.wav"
|
| 274 |
+
speech, sample_rate = soundfile.read(wav_file)
|
| 275 |
+
chunk_stride = int(chunk_size * sample_rate / 1000)
|
| 276 |
+
|
| 277 |
+
cache = {}
|
| 278 |
+
total_chunk_num = int(len((speech)-1)/chunk_stride+1)
|
| 279 |
+
for i in range(total_chunk_num):
|
| 280 |
+
speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
|
| 281 |
+
is_final = i == total_chunk_num - 1
|
| 282 |
+
res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size)
|
| 283 |
+
if len(res[0]["value"]):
|
| 284 |
+
print(res)
|
| 285 |
+
```
|
| 286 |
+
|
| 287 |
+
#### 标点恢复
|
| 288 |
+
```python
|
| 289 |
+
from funasr import AutoModel
|
| 290 |
+
|
| 291 |
+
model = AutoModel(model="ct-punc", model_revision="v2.0.4")
|
| 292 |
+
|
| 293 |
+
res = model.generate(input="那今天的会就到这里吧 happy new year 明年见")
|
| 294 |
+
print(res)
|
| 295 |
+
```
|
| 296 |
+
|
| 297 |
+
#### 时间戳预测
|
| 298 |
+
```python
|
| 299 |
+
from funasr import AutoModel
|
| 300 |
+
|
| 301 |
+
model = AutoModel(model="fa-zh", model_revision="v2.0.4")
|
| 302 |
+
|
| 303 |
+
wav_file = f"{model.model_path}/example/asr_example.wav"
|
| 304 |
+
text_file = f"{model.model_path}/example/text.txt"
|
| 305 |
+
res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))
|
| 306 |
+
print(res)
|
| 307 |
+
```
|
| 308 |
+
|
| 309 |
+
更多详细用法([示例](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining))
|
| 310 |
+
|
| 311 |
+
|
| 312 |
+
## 微调
|
| 313 |
+
|
| 314 |
+
详细用法([示例](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining))
|
| 315 |
+
|
| 316 |
+
|
| 317 |
+
## Benchmark
|
| 318 |
+
结合大数据、大模型优化的Paraformer在一序列语音识别的benchmark上获得当前SOTA的效果,以下展示学术数据集AISHELL-1、AISHELL-2、WenetSpeech,公开评测项目SpeechIO TIOBE白盒测试场景的效果。在学术界常用的中文语音识别评测任务中,其表现远远超于目前公开发表论文中的结果,远好于单独封闭数据集上的模型。
|
| 319 |
+
|
| 320 |
+
### AISHELL-1
|
| 321 |
+
|
| 322 |
+
| AISHELL-1 test | w/o LM | w/ LM |
|
| 323 |
+
|:------------------------------------------------:|:-------------------------------------:|:-------------------------------------:|
|
| 324 |
+
| <div style="width: 150pt">Espnet</div> | <div style="width: 150pt">4.90</div> | <div style="width: 150pt">4.70</div> |
|
| 325 |
+
| <div style="width: 150pt">Wenet</div> | <div style="width: 150pt">4.61</div> | <div style="width: 150pt">4.36</div> |
|
| 326 |
+
| <div style="width: 150pt">K2</div> | <div style="width: 150pt">-</div> | <div style="width: 150pt">4.26</div> |
|
| 327 |
+
| <div style="width: 150pt">Blockformer</div> | <div style="width: 150pt">4.29</div> | <div style="width: 150pt">4.05</div> |
|
| 328 |
+
| <div style="width: 150pt">Paraformer-large</div> | <div style="width: 150pt">1.95</div> | <div style="width: 150pt">1.68</div> |
|
| 329 |
+
|
| 330 |
+
### AISHELL-2
|
| 331 |
+
|
| 332 |
+
| | dev_ios| test_android| test_ios|test_mic|
|
| 333 |
+
|:-------------------------------------------------:|:-------------------------------------:|:-------------------------------------:|:------------------------------------:|:------------------------------------:|
|
| 334 |
+
| <div style="width: 150pt">Espnet</div> | <div style="width: 70pt">5.40</div> |<div style="width: 70pt">6.10</div> |<div style="width: 70pt">5.70</div> |<div style="width: 70pt">6.10</div> |
|
| 335 |
+
| <div style="width: 150pt">WeNet</div> | <div style="width: 70pt">-</div> |<div style="width: 70pt">-</div> |<div style="width: 70pt">5.39</div> |<div style="width: 70pt">-</div> |
|
| 336 |
+
| <div style="width: 150pt">Paraformer-large</div> | <div style="width: 70pt">2.80</div> |<div style="width: 70pt">3.13</div> |<div style="width: 70pt">2.85</div> |<div style="width: 70pt">3.06</div> |
|
| 337 |
+
|
| 338 |
+
|
| 339 |
+
### Wenetspeech
|
| 340 |
+
|
| 341 |
+
| | dev| test_meeting| test_net|
|
| 342 |
+
|:-------------------------------------------------:|:-------------------------------------:|:-------------------------------------:|:------------------------------------:|
|
| 343 |
+
| <div style="width: 150pt">Espnet</div> | <div style="width: 100pt">9.70</div> |<div style="width: 100pt">15.90</div> |<div style="width: 100pt">8.80</div> |
|
| 344 |
+
| <div style="width: 150pt">WeNet</div> | <div style="width: 100pt">8.60</div> |<div style="width: 100pt">17.34</div> |<div style="width: 100pt">9.26</div> |
|
| 345 |
+
| <div style="width: 150pt">K2</div> | <div style="width: 100pt">7.76</div> |<div style="width: 100pt">13.41</div> |<div style="width: 100pt">8.71</div> |
|
| 346 |
+
| <div style="width: 150pt">Paraformer-large</div> | <div style="width: 100pt">3.57</div> |<div style="width: 100pt">6.97</div> |<div style="width: 100pt">6.74</div> |
|
| 347 |
+
|
| 348 |
+
### SpeechIO TIOBE
|
| 349 |
+
|
| 350 |
+
Paraformer-large模型结合Transformer-LM模型做shallow fusion,在公开评测项目SpeechIO TIOBE白盒测试场景上获得当前SOTA的效果,目前[Transformer-LM模型](https://modelscope.cn/models/damo/speech_transformer_lm_zh-cn-common-vocab8404-pytorch/summary)已在ModelScope上开源,以下展示SpeechIO TIOBE白盒测试场景without LM、with Transformer-LM的效果:
|
| 351 |
+
|
| 352 |
+
- Decode config w/o LM:
|
| 353 |
+
- Decode without LM
|
| 354 |
+
- Beam size: 1
|
| 355 |
+
- Decode config w/ LM:
|
| 356 |
+
- Decode with [Transformer-LM](https://modelscope.cn/models/damo/speech_transformer_lm_zh-cn-common-vocab8404-pytorch/summary)
|
| 357 |
+
- Beam size: 10
|
| 358 |
+
- LM weight: 0.15
|
| 359 |
+
|
| 360 |
+
| testset | w/o LM | w/ LM |
|
| 361 |
+
|:------------------:|:----:|:----:|
|
| 362 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00001</div>| <div style="width: 150pt">0.49</div> | <div style="width: 150pt">0.35</div> |
|
| 363 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00002</div>| <div style="width: 150pt">3.23</div> | <div style="width: 150pt">2.86</div> |
|
| 364 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00003</div>| <div style="width: 150pt">1.13</div> | <div style="width: 150pt">0.80</div> |
|
| 365 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00004</div>| <div style="width: 150pt">1.33</div> | <div style="width: 150pt">1.10</div> |
|
| 366 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00005</div>| <div style="width: 150pt">1.41</div> | <div style="width: 150pt">1.18</div> |
|
| 367 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00006</div>| <div style="width: 150pt">5.25</div> | <div style="width: 150pt">4.85</div> |
|
| 368 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00007</div>| <div style="width: 150pt">5.51</div> | <div style="width: 150pt">4.97</div> |
|
| 369 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00008</div>| <div style="width: 150pt">3.69</div> | <div style="width: 150pt">3.18</div> |
|
| 370 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH00009</div>| <div style="width: 150pt">3.02</div> | <div style="width: 150pt">2.78</div> |
|
| 371 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH000010</div>| <div style="width: 150pt">3.35</div> | <div style="width: 150pt">2.99</div> |
|
| 372 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH000011</div>| <div style="width: 150pt">1.54</div> | <div style="width: 150pt">1.25</div> |
|
| 373 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH000012</div>| <div style="width: 150pt">2.06</div> | <div style="width: 150pt">1.68</div> |
|
| 374 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH000013</div>| <div style="width: 150pt">2.57</div> | <div style="width: 150pt">2.25</div> |
|
| 375 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH000014</div>| <div style="width: 150pt">3.86</div> | <div style="width: 150pt">3.08</div> |
|
| 376 |
+
|<div style="width: 200pt">SPEECHIO_ASR_ZH000015</div>| <div style="width: 150pt">3.34</div> | <div style="width: 150pt">2.67</div> |
|
| 377 |
+
|
| 378 |
+
|
| 379 |
+
## 使用方式以及适用范围
|
| 380 |
+
|
| 381 |
+
运行范围
|
| 382 |
+
- 支持Linux-x86_64、Mac和Windows运行。
|
| 383 |
+
|
| 384 |
+
使用方式
|
| 385 |
+
- 直接推理:可以直接对输入音频进行解码,输出目标文字。
|
| 386 |
+
- 微调:加载训练好的模型,采用私有或者开源数据进行模型训练。
|
| 387 |
+
|
| 388 |
+
使用范围与目标场景
|
| 389 |
+
- 适合与离线语音识别场景,如录音文件转写,配合GPU推理效果更加,推荐输入语音时长在20s以下,若想解码长音频,推荐使用[Paraformer-large长音频模型](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary),集成VAD、ASR、标点与时间戳功能,可直接对时长为数小时音频进行识别,并输出带标点文字与时间戳。
|
| 390 |
+
|
| 391 |
+
|
| 392 |
+
## 模型局限性以及可能的偏差
|
| 393 |
+
|
| 394 |
+
考虑到特征提取流程和工具以及训练工具差异,会对CER的数据带来一定的差异(<0.1%),推理GPU环境差异导致的RTF数值差异。
|
| 395 |
+
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
## 相关论文以及引用信息
|
| 399 |
+
|
| 400 |
+
```BibTeX
|
| 401 |
+
@inproceedings{gao2022paraformer,
|
| 402 |
+
title={Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition},
|
| 403 |
+
author={Gao, Zhifu and Zhang, Shiliang and McLoughlin, Ian and Yan, Zhijie},
|
| 404 |
+
booktitle={INTERSPEECH},
|
| 405 |
+
year={2022}
|
| 406 |
+
}
|
| 407 |
+
```
|