README.md

metadata

license: other
license_name: funasr-model-license
license_link: https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE
library_name: transformers
base_model:
  - FunAudioLLM/SenseVoiceSmall
tags:
  - duplexjev
  - audio
  - asr-encoder
  - sensevoice

SenseVoice-Small-Encoder

The audio encoder of FunAudioLLM/SenseVoiceSmall, packaged as a standalone transformers model (trust_remote_code=True) so that it can feed a frozen LLM through a DuplexJev connector. It needs no FunASR installation. The weights are those of SenseVoiceSmall, unchanged: the SANM encoder (50 + 20 blocks, d = 512), the task-query embeddings and the CMVN statistics from am.mvn. The CTC head is not included.

What the model does inside forward:

1. SenseVoice's frontend: Kaldi fbank (80 bins, 25 ms / 10 ms, Hamming, dither 0), LFR (7 frames stacked, stride 6), CMVN.
2. The four task queries SenseVoice prepends (language = auto, event, emotion, text-norm = woitn).
3. The encoder. The four query frames are dropped from the output, so one output frame covers 60 ms of audio.

The feature extractor only cuts 16 kHz audio into 10 ms frames of raw samples; the encoder computes the features. This keeps the Ultravox processor's length arithmetic unchanged (encoder_ds_factor = 6).

Port check: greedy CTC decoding with SenseVoiceSmall's own CTC head on top of this encoder reproduces correct transcripts of Chinese and English test clips, in fp32 and bf16.

Requires torchaudio (for the Kaldi fbank): pip install torchaudio.

Used by the DuplexJev-B[-Para]-SenseVoice-Small-* connectors; Decider.from_pretrained("adventists-ai/DuplexJev-...") fetches it automatically. Source of the port: github.com/adventists-ai/duplexjev.

License and attribution

The weights are SenseVoiceSmall by Alibaba Group (Tongyi Lab / FunAudioLLM), redistributed under the FunASR Model Open Source License v1.1 (copy in MODEL_LICENSE), which requires attributing the source and author and keeping the model name. The model is provided by its authors for reference and learning; see the licence for the full terms. The SANM layer code is copied from FunASR (MIT). The packaging code (feature extractor, wrapper) is Apache-2.0. Please cite SenseVoice / FunAudioLLM when you use it.