Hugging Face's logo

Veeda241
/
emora-audio-emotion2vec

Audio Classification
English
emotion2vec_classifier
speech-emotion-recognition
emotion2vec
ravdess
multimodal
emotion-aware-agent
Model card Files
xet
 Community
emora-audio-emotion2vec / README.md
Veeda241's picture
Veeda241
Add license metadata (apache-2.0, matching sibling EMORA branches)
48b6e72 verified
preview code
|
Raw History Blame Contribute Delete
7.69 kB
---
license: apache-2.0
pipeline_tag: audio-classification
language:
- en
tags:
- audio-classification
- speech-emotion-recognition
- emotion2vec
- ravdess
- multimodal
- emotion-aware-agent
---
# emotion2vec classification head for 6-class speech emotion recognition
A small trained classifier that sits on top of the frozen [`iic/emotion2vec_base_finetuned`](https://modelscope.cn/models/iic/emotion2vec_base_finetuned) speech encoder. Given a speech clip, it predicts one of six emotions: **angry, disgust, fearful, happy, sad, surprised**.
It is the audio branch of the *LLM-Based Multimodal Behaviour-Change Support* project, where speech emotion is combined with text and facial-expression signals before a language model writes a reply.
> **This repository holds only the head (about 0.2M parameters). The backbone is not included.** You load the backbone separately and attach this head to it.
## At a glance
| | |
|---|---|
| Task | Utterance-level speech emotion classification |
| Classes | 6: angry, disgust, fearful, happy, sad, surprised |
| Input | Speech waveform, 16 kHz, mono |
| Backbone | `iic/emotion2vec_base_finetuned` (frozen, 768-dimensional frame embeddings) |
| Trainable parameters | 198,918 (the head only) |
| Training data | RAVDESS, 768 training clips |
| Test accuracy / macro F1 | **92.7% / 0.927** on 192 held-out clips from unseen speakers |
| Weights file | `model_state_dict.pt` |
## How it works
```
waveform (16 kHz, mono)
-> emotion2vec backbone (frozen) 768 numbers per frame
-> mean over time frames 768
-> Linear(768 -> 256)
-> LayerNorm(256)
-> GELU
-> Dropout(0.1)
-> Linear(256 -> 6) 6 logits -> softmax
```
| Layer | Parameters |
|---|---|
| Linear 768 to 256 | 196,864 |
| LayerNorm 256 | 512 |
| Linear 256 to 6 | 1,542 |
| **Total** | **198,918** |
Label ids: `0 angry, 1 disgust, 2 fearful, 3 happy, 4 sad, 5 surprised`.
The saved keys carry a `classifier.` prefix. Strip it before loading into a bare head, as shown below.
## How to use
Install the dependencies:
```bash
pip install torch funasr huggingface_hub
```
```python
import torch
import torch.nn as nn
from funasr import AutoModel
from huggingface_hub import hf_hub_download
LABELS = ["angry", "disgust", "fearful", "happy", "sad", "surprised"]
def build_head(hidden=768, num_labels=6):
return nn.Sequential(
nn.Linear(hidden, 256), nn.LayerNorm(256), nn.GELU(),
nn.Dropout(0.1), nn.Linear(256, num_labels),
)
# 1. Backbone (downloaded at runtime)
backbone = AutoModel(model="iic/emotion2vec_base_finetuned")
# 2. Head: load the weights and strip the "classifier." prefix from the keys
head = build_head()
path = hf_hub_download(repo_id="<your-username>/<this-repo>", filename="model_state_dict.pt")
state = torch.load(path, map_location="cpu")
state = {k.removeprefix("classifier."): v for k, v in state.items()}
head.load_state_dict(state)
head.eval()
# 3. Predict
out = backbone.generate("clip.wav", granularity="frame", extract_embedding=True)
frames = torch.tensor(out[0]["feats"]) # (time, 768)
probs = head(frames.mean(dim=0)).softmax(-1) # mean over time, as in training
print({l: round(float(p), 3) for l, p in zip(LABELS, probs)})
```
The reference implementation is `src/pipeline/audio_pipeline.py` in the source project. If it differs from the snippet above (for example in how the embedding is extracted), follow the project code.
## Training
| Setting | Value |
|---|---|
| Dataset | RAVDESS, 6 classes (neutral and calm excluded) |
| Samples | 1,152 total: 768 train, 192 validation, 192 test |
| Split | By actor, so no speaker appears in two splits: actors 1 to 16 train, 17 to 20 validation, 21 to 24 test |
| Epochs | 10 |
| Batch size | 8 |
| Learning rate | 2e-5 |
| Optimizer | AdamW, weight decay 0.01 |
| Gradient clipping | Norm 1.0 |
| Mixed precision | Yes |
| Seed | 42 |
| Loss | Cross-entropy |
| Checkpoint choice | Lowest validation loss (epoch 2) |
| Training time | 232 seconds |
| Hardware | Single RTX 3050 6 GB laptop GPU; the backbone runs on the CPU |
Only the head is trained. The backbone embeddings are used as fixed features, which is why training is fast and overfits less than full fine-tuning.
## Evaluation
Test set: 192 clips from four unseen actors, 32 clips per class.
| Metric | Value |
|---|---|
| Accuracy | 92.71% |
| Macro F1 | 0.9271 |
| Weighted F1 | 0.9271 |
| Emotion | Precision | Recall | F1 |
|---|---|---|---|
| angry | 1.000 | 1.000 | 1.000 |
| disgust | 0.889 | 1.000 | 0.941 |
| fearful | 0.929 | 0.813 | 0.867 |
| happy | 1.000 | 0.938 | 0.968 |
| sad | 0.794 | 0.844 | 0.818 |
| surprised | 0.969 | 0.969 | 0.969 |
Most errors: 6 fearful clips predicted as sad, and 4 sad clips predicted as disgust.
### Comparison with other speech models
All three were trained with identical settings on the same split.
| Model | What was trained | Test accuracy | Macro F1 |
|---|---|---|---|
| wav2vec2-base | Whole model | 68.2% | 0.669 |
| WavLM-base | Whole model | 72.4% | 0.711 |
| **emotion2vec + this head** | Head only | **92.7%** | **0.927** |
## Intended use
- Research and prototyping of emotion-aware conversational agents.
- One input signal among several: it is designed to be combined with text and face signals, not to be the only source of truth.
## Out-of-scope use
- Diagnosing or inferring any mental-health or clinical condition from speech. The labels describe expressed emotion in short clips, not a person's health.
- Decisions that affect people, such as hiring, insurance, policing or education.
- Recording or analysing someone's voice without their knowledge and consent.
## Limitations and bias
- **Acted speech.** RAVDESS is made of professional actors reading fixed sentences. Natural, spontaneous speech is harder, and the 92.7% figure should not be expected to carry over.
- **Small and narrow data.** 24 speakers, one language (English), six emotions. Accents, other languages, age groups, noisy rooms and phone-quality audio have not been tested.
- **Closed label set.** Neutral and calm were removed, so the model must pick one of six emotions even when none fits.
- **Backbone overlap not checked.** The backbone checkpoint is itself fine-tuned for emotion recognition. We have not verified whether its fine-tuning data overlaps RAVDESS, so the score may overstate how well the model generalises. Testing on a second dataset is planned.
- **Backbone access.** The backbone is downloaded at runtime and may be gated or rate-limited depending on your account and token.
- Audio quality, speaker identity and language all affect predictions.
## Files
| File | Description |
|---|---|
| `model_state_dict.pt` | Head weights (keys prefixed with `classifier.`) |
| `README.md` | This model card |
## Source
- Weights: `models/audio/runs_emotion2vec/model_state_dict.pt`
- Training script: `models/audio/train.py`
- Inference pipeline: `src/pipeline/audio_pipeline.py`
## Citation
If you use this head, please also cite the backbone and the dataset.
```bibtex
@article{ma2023emotion2vec,
title = {emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation},
author = {Ma, Ziyang and Zheng, Zhisheng and Ye, Jiaxin and Li, Jinchao and Gao, Zhifu and Zhang, Shiliang and Chen, Xie},
journal = {arXiv preprint arXiv:2312.15185},
year = {2023}
}
@article{livingstone2018ravdess,
title = {The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)},
author = {Livingstone, Steven R. and Russo, Frank A.},
journal = {PLOS ONE},
volume = {13},
number = {5},
pages = {e0196391},
year = {2018}
}
```