| --- |
| license: other |
| license_name: funasr-model-license |
| license_link: https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE |
| library_name: transformers |
| base_model: |
| - FunAudioLLM/SenseVoiceSmall |
| tags: |
| - duplexjev |
| - audio |
| - asr-encoder |
| - sensevoice |
| --- |
| |
| # SenseVoice-Small-Encoder |
|
|
| The audio encoder of [FunAudioLLM/SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall), packaged as a |
| standalone `transformers` model (`trust_remote_code=True`) so that it can feed a frozen LLM through a DuplexJev |
| connector. It needs no FunASR installation. The weights are those of SenseVoiceSmall, unchanged: the SANM encoder |
| (50 + 20 blocks, d = 512), the task-query embeddings and the CMVN statistics from `am.mvn`. The CTC head is not included. |
|
|
| What the model does inside `forward`: |
|
|
| 1. SenseVoice's frontend: Kaldi fbank (80 bins, 25 ms / 10 ms, Hamming, dither 0), LFR (7 frames stacked, stride 6), |
| CMVN. |
| 2. The four task queries SenseVoice prepends (language = auto, event, emotion, text-norm = woitn). |
| 3. The encoder. The four query frames are dropped from the output, so one output frame covers 60 ms of audio. |
|
|
| The feature extractor only cuts 16 kHz audio into 10 ms frames of raw samples; the encoder computes the features. This |
| keeps the Ultravox processor's length arithmetic unchanged (`encoder_ds_factor = 6`). |
|
|
| Port check: greedy CTC decoding with SenseVoiceSmall's own CTC head on top of this encoder reproduces correct |
| transcripts of Chinese and English test clips, in fp32 and bf16. |
|
|
| Requires `torchaudio` (for the Kaldi fbank): `pip install torchaudio`. |
|
|
| Used by the `DuplexJev-B[-Para]-SenseVoice-Small-*` connectors; `Decider.from_pretrained("adventists-ai/DuplexJev-...")` |
| fetches it automatically. Source of the port: [github.com/adventists-ai/duplexjev](https://github.com/adventists-ai/duplexjev). |
|
|
| ## License and attribution |
|
|
| The weights are **SenseVoiceSmall by Alibaba Group (Tongyi Lab / FunAudioLLM)**, redistributed under the |
| [FunASR Model Open Source License v1.1](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE) (copy in |
| `MODEL_LICENSE`), which requires attributing the source and author and keeping the model name. The model is provided |
| by its authors for reference and learning; see the licence for the full terms. The SANM layer code is copied from |
| [FunASR](https://github.com/modelscope/FunASR) (MIT). The packaging code (feature extractor, wrapper) is Apache-2.0. |
| Please cite SenseVoice / FunAudioLLM when you use it. |
|
|
