| --- |
| language: |
| - zh |
| - en |
| license: apache-2.0 |
| pipeline_tag: automatic-speech-recognition |
| base_model: FireRedTeam/FireRedASR2-AED |
| tags: |
| - ctranslate2 |
| - audio |
| - asr |
| - firered |
| --- |
| |
| # FireRedASR2-AED, converted to CTranslate2 (float32) |
|
|
| This is [FireRedTeam/FireRedASR2-AED](https://huggingface.co/FireRedTeam/FireRedASR2-AED) converted to the |
| CTranslate2 format, without quantization (float32 weights). The original model and its license |
| (Apache-2.0) belong to the FireRedTeam; see the [FireRedASR2S repository](https://github.com/FireRedTeam/FireRedASR2S). |
| Nothing about the model itself was retrained or changed. |
|
|
| The directory contains `model.bin`, `config.json`, `vocabulary.json`, the SentencePiece model `bpe.model` and the |
| CMVN statistics `cmvn.ark`. |
|
|
| ## Requirements |
|
|
| Stock CTranslate2 and faster-whisper cannot load this model. FireRed is not a Whisper model: it has a Conformer |
| encoder with Transformer-XL relative-position attention and its own front end and tokenizer. Loading needs a |
| CTranslate2 build that includes the `FireRed` model class (the voxrun-ctranslate2 fork, with FireRed support). |
| That support is not part of a released wheel yet. |
|
|
| ## Usage |
|
|
| ```python |
| import ctranslate2 |
| from ctranslate2.models.firered_features import FireRedFeatureExtractor |
| from ctranslate2.models.firered_tokenizer import FireRedTokenizer |
| |
| path = "FireRedASR2-AED-ct2" # this repository, downloaded |
| model = ctranslate2.models.FireRed(path, device="cuda", compute_type="float16") |
| extractor = FireRedFeatureExtractor(path) # needs kaldi-native-fbank |
| tokenizer = FireRedTokenizer(path) # needs sentencepiece |
| |
| samples, rate = extractor.read_wav("audio.wav") # 16 kHz mono |
| features, lengths = extractor([samples], rate) |
| result = model.transcribe(ctranslate2.StorageView.from_array(features), lengths)[0] |
| print(tokenizer.detokenize(result.sequences_ids[0])) |
| ``` |
|
|
| The weights are float32 (4.7 GB). Choose the precision at load time with `compute_type`: `float16`, `bfloat16`, |
| `int8_float16`, `int8_bfloat16` and CPU `int8` all work. Output is Simplified Chinese. |
|
|
| ## Accuracy (FLEURS test, CTranslate2 float32, beam 3) |
|
|
| Character error rate after removing case, spaces and punctuation: Mandarin 10.21 %, Cantonese 8.46 % (scored |
| against Simplified Chinese). Float16 and bfloat16 stay within 0.4 points of these numbers. On the CTranslate2 |
| side the token IDs of float32 decoding match the upstream PyTorch output on the clips that were compared. |
|
|
