| --- |
| license: apache-2.0 |
| pipeline_tag: audio-to-audio |
| tags: |
| - speech |
| - audio |
| - codec |
| - speech-codec |
| - whisper |
| - low-bitrate |
| - audio-compression |
| language: |
| - en |
| datasets: |
| - librispeech |
| library_name: pytorch |
| --- |
| |
|
|
| <div align="center"> |
|
|
| # 🎙️ SimWhisper-Codec |
|
|
| ### Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding |
|
|
| <p> |
| <a href="https://zhangxinwhut.github.io/SimWhisper-Codec/"><img src="https://img.shields.io/badge/🎧_Demo-Online-brightgreen" alt="Demo"></a> |
| <a href="https://arxiv.org/pdf/2510.20504v2"><img src="https://img.shields.io/badge/Paper-Arxiv-red" alt="paper"></a> |
| <a href="https://huggingface.co/xxx123456/SimWhisper_Codec"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model%20Page-yellow" alt="Hugging Face"></a> |
| <a href="https://github.com/ZhangXinWhut/SimWhisper-Codec"><img src="https://img.shields.io/badge/GitHub-Code-black?logo=github" alt="GitHub"></a> |
| </p> |
|
|
| *A semantic-first speech codec that achieves superior performance through architectural simplification rather than complex supervision.* |
|
|
| </div> |
|
|
| --- |
|
|
| ## ✨ Highlights |
|
|
| - 🚀 **low Bitrate**: Only **1.1 kbps** at 16 kHz sampling rate |
| - 🔊 **High Quality Speech Reconstruction**: Achieving UTMOS 4.00 WER 2.75 (hubert-large-ls960-ft) sim 0.83 (wavlm_large_finetune) stoi 0.93 pesq-nb 3.29 pesq-wb 2.72 on librispeech-test-clean reconstruction (gt: WER 2.16 UTMOS 4.09) |
| - 🧊 **Frozen Encoder**: No fine-tuning of Whisper encoder required |
| - ⚡ **Simple & Efficient**: Architectural simplification over complex supervision |
|
|
| ## 📊 Performance |
|
|
| | Model | Bitrate | WER ↓ | PESQ-NB ↑ | PESQ-WB ↑ | STOI ↑ | SIM ↑ | UTMOS ↑ | |
| |:------|:-------:|:-----:|:---------:|:---------:|:------:|:-----:|:-------:| |
| | XCodec2.0 | 0.8 kbps | 2.61 | 3.04 | 2.43 | 0.92 | 0.82 | **4.13** | |
| | XY-Tokenizer | 1.0 kbps | **2.46** | 3.00 | 2.41 | 0.91 | **0.84** | 3.98 | |
| | **SimWhisper-Codec** | 1.1 kbps | 2.75 | **3.29** | **2.72** | **0.93** | 0.83 | 4.00 | |
|
|
| *Evaluated on LibriSpeech test-clean* |
|
|
| ## 🚀 Quick Start |
|
|
| ### Installation |
|
|
| ```bash |
| # Clone repository |
| git clone https://github.com/ZhangXinWhut/SimWhisper-Codec.git && cd SimWhisper-Codec |
| |
| # Create and activate conda environment |
| conda create -n swcodec python=3.10 -y && conda activate swcodec |
| |
| # Install dependencies |
| pip install -r requirements.txt |
| ``` |
|
|
| ## Available Models 🗂️ |
|
|
| | Model Name | Hugging Face | Training Data | |
| |:----------:|:-------------:|:---------------:| |
| | SimWhisper-Codec | [🤗](https://huggingface.co/xxx123456/SimWhisper_Codec) | LibriSpeech | |
|
|
|
|
| ### Download Model Weights |
|
|
| You need to download the SimWhisper-Codec model weights. You can find the weights in the [SimWhisper-Codec Hugging Face repository](https://huggingface.co/xxx123456/SimWhisper_Codec). |
|
|
| ```bash |
| mkdir -p ./weights && huggingface-cli download xxx123456/SimWhisper_Codec SimWhisperCodec.pt --local-dir ./weights/ |
| ``` |
|
|
| ### Inference |
|
|
| ```python |
| python inference.py --input_dir /path/to/LibriSpeech/test-clean |
| ``` |
|
|
| The reconstructed audio files will be available in the `output_wavs/` directory. |
|
|
| ## 🙏 Acknowledgements |
|
|
| Our codebase builds upon the [XY-Tokenizer](https://github.com/gyt1145028706/XY-Tokenizer). We thank the authors for their excellent work. |
|
|
| ## 📝 Citation |
|
|
| If you find this work useful in your research, please cite our paper: |
|
|
| ``` |
| @misc{zhang2025speakingclearlysimplifiedwhisperbased, |
| title={Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding}, |
| author={Xin Zhang and Lin Li and Xiangni Lu and Jianquan Liu and Kong Aik Lee}, |
| year={2025}, |
| eprint={2510.20504}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.SD}, |
| url={https://arxiv.org/abs/2510.20504}, |
| } |
| ``` |
|
|
