README.md

---
license: apache-2.0
---
# BLSP-Emo: Towards Empathetic Large Speech-Language Models

Chen Wang, Minpeng Liao, Zhongqiang Huang,Junhong Wu, Chenqing Zong, Jiajun Zhang

**Institute of Automation, Chinese Academy of Sciences**

**Alibaba Group**

<a href='https://www.modelscope.cn/studios/Decaderan/Blsp-Qwen-7B-Demo/summary'><img src='https://img.shields.io/badge/ModelScope-Demo-blueviolet'></a>
<a href=''><img src='https://img.shields.io/badge/ModelScope-Checkpoint-blueviolet'></a>
<a href=''><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Checkpoint-blue'></a> <a href='https://cwang621.github.io/blsp-emo.github.io'><img src='https://img.shields.io/badge/Project-Page-Green'></a>  <a href='https://arxiv.org/abs/2406.03872'><img src='https://img.shields.io/badge/Paper-Arxiv-red'> </a>


## Introduction

* BLSP-Emo is designed to enable an end-to-end speech-language model to understand emotions in speech and generate empathetic responses, using only existing ASR and SER data. 
* BLSP-Emo is built based on Whisper-large-v2 and Qwen-7B-Chat.

![architecture](figures/emotion_blsp.png)

## Example

![Demo](figures/emo-motivation.png)



More examples can be found in the [project page](https://cwang621.github.io/blsp-emo.github.io). You can also try our model online at [modelscope](https://www.modelscope.cn/studios/Decaderan/Blsp-Qwen-7B-Demo/summary).



## License
* The license of our project is [Apache License 2.0]()
* Our models are based on Qwen and Whisper. If you want to use our models, please do not violate the [MIT License](https://github.com/openai/whisper/blob/main/LICENSE) of whisper and the [License](https://github.com/QwenLM/Qwen/blob/main/LICENSE) of Qwen

## Citation
If you find our project useful, hope you can star our repo and cite our paper as follows:
```
@misc{wang2024blspemo,
      title={BLSP-Emo: Towards Empathetic Large Speech-Language Models}, 
      author={Chen Wang and Minpeng Liao and Zhongqiang Huang and Junhong Wu and Chengqing Zong and Jiajun Zhang},
      year={2024},
      eprint={2406.03872},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```