| --- |
| library_name: transformers |
| license: apache-2.0 |
| language: |
| - en |
| base_model: facebook/wav2vec2-large-robust-ft-libri-960h |
| datasets: |
| - nguyenvulebinh/AVYT |
| - nguyenvulebinh/AVCocktail |
| metrics: |
| - wer |
| tags: |
| - custom_code |
| - msp |
| - msp_audio |
| - audio |
| - ctc |
| - speech-recognition |
| - ASR |
| --- |
| |
| # MSP-ASR |
|
|
| MSP-ASR is the audio model in [Multimodal Speech Perception (MSP)](https://github.com/Mahmood-Anaam/msp). It fine-tunes a Wav2Vec2 encoder with a CTC head for English speech recognition and supports greedy decoding or the companion 3-gram language model. |
|
|
| | Property | Value | |
| |---|---| |
| | Input | Mono, 16 kHz audio | |
| | Architecture | Wav2Vec2 encoder + CTC head | |
| | Base checkpoint | `facebook/wav2vec2-large-robust-ft-libri-960h` | |
| | Demo | [MSP-ASR](https://huggingface.co/spaces/MahmoodAnaam/MSP-ASR) | |
|
|
| MSP-ASR also supplies the audio encoder to MSP-AVSR. There, its representations participate in both directions of the bidirectional cross-attention fusion block; the standalone checkpoint remains audio-only. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCTC, AutoProcessor |
| |
| model_id = "MahmoodAnaam/MSP-ASR" |
| processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCTC.from_pretrained(model_id, trust_remote_code=True).eval() |
| |
| inputs = processor(audio="sample.wav", return_tensors="pt") |
| with torch.inference_mode(): |
| logits = model(**inputs).logits |
| |
| text = processor.tokenizer.batch_decode(logits.argmax(dim=-1))[0] |
| print(text) |
| ``` |
|
|
| For beam search, install `pyctcdecode` and `kenlm`, then load `MahmoodAnaam/MSP-Processor-With-LM` and call `batch_decode(logits.cpu().numpy())`. Pin Hub revisions when loading custom code in controlled environments. |
|
|
| ## Training |
|
|
| | Setting | Value | |
| |---|---:| |
| | Maximum steps | 40,000 | |
| | Learning rate | 1e-4 | |
| | Train / eval batch size per device | 16 / 16 | |
| | Gradient accumulation | 2 | |
| | Scheduler | Cosine; 1,000 warmup steps | |
| | Precision | bfloat16 | |
|
|
| The selected validation checkpoint recorded loss **0.3481** and WER **20.40%**. The recorded stack was Transformers 5.10.2, PyTorch 2.10.0 with ROCm 7.2.4, Datasets 4.0.0, and Tokenizers 0.22.2. |
|
|
| ## Evaluation |
|
|
| The following tables report **MSP-ASR only**. Results use `MSP-Processor-With-LM`; WER is a percentage and lower is better. |
|
|
| ### LRS2 |
|
|
| | Evaluation | WER | |
| |---|---:| |
| | Clean test | 6.30 | |
| | Clean + eight noisy conditions, macro average | 46.34 | |
|
|
| | Interferer | -5 dB | 0 dB | 5 dB | 10 dB | |
| |---:|---:|---:|---:|---:| |
| | 1 | 90.22 | 42.76 | 47.40 | 22.52 | |
| | 2 | 92.99 | 42.86 | 48.91 | 23.09 | |
|
|
| ### AVCocktail |
|
|
| | Segmentation | WER | |
| |---|---:| |
| | Active-speaker-detection chunks | 77.62 | |
| | Fixed 10-second chunks | 127.36 | |
| | Gold chunks | 74.37 | |
|
|
|
|
| ## Resources |
|
|
| - [Source and training configuration](https://github.com/Mahmood-Anaam/msp) |
| - [MSP-ASR demo](https://huggingface.co/spaces/MahmoodAnaam/MSP-ASR) |
| - [3-gram LM processor](https://huggingface.co/MahmoodAnaam/MSP-Processor-With-LM) |
