---
license: apache-2.0
tags:
- speech-emotion-recognition
- emotion2vec
- speech-to-speech-translation
- tts
- conditioning
library_name: funasr
---

# Surgical Emotion Module — emotion2vec_plus_base @ 12.5 Hz

A surgically modified **emotion2vec_plus_base** that outputs **continuous, frame-level
emotion vectors at 12.5 Hz** (80 ms/frame) instead of a single pooled utterance
vector — designed as the emotion-conditioning front-end of a Speech-to-Speech
Translation (S2ST) system built on top of **dots.tts** (its LLM token rate is
6.25 Hz, so exactly **2 emotion frames align to every LLM token**).

## What the surgery changes

| | stock emotion2vec_plus_base | this module |
|---|---|---|
| output | 1 pooled 768-d utterance vector | per-frame **768-d continuous vectors @ 12.5 Hz** |
| emotion probs | utterance-level, 9 classes | **per-frame** 9 classes + utterance-level |
| intensity | none | **continuous [0,1] per frame** |
| dynamics | discarded by pooling | preserved (within-word intensity changes) |

Architecture of the implant (3.18 M trainable params, the 93.8 M backbone stays frozen):

```
frozen emotion2vec_plus_base backbone (50 Hz frame states, 768-d)
  -> LayerNorm -> Linear(768->512)
  -> 3x depthwise-separable conv fusion (k=9, dilation 1/2/4)
  -> grouped soft-attention pooling over G=4 frames   (50 Hz -> 12.5 Hz)
  -> LayerNorm -> Linear(512->768)                     (continuous emotion vectors)
  -> heads: emotion logits (9)  +  intensity (sigmoid)
```

Intensity is distilled from a confidence-gated expression-energy target
(sqrt(max-prob) x utterance-normalized RMS envelope), which gives real temporal
dynamics instead of the saturated confidence proxy.

## Outputs

```python
out = enc(wav_16k, wav_lengths)
out["frame_embs"]      # (B, T, 768) continuous emotion vectors @ 12.5 Hz
out["frame_probs"]     # (B, T, 9)   per-frame emotion probabilities
out["frame_intensity"] # (B, T, 1)   per-frame intensity in [0,1]
out["utt_emb"]         # (B, 768)    pooled utterance embedding
out["utt_probs"]       # (B, 9)
out["frame_lengths"]   # (B,)
# T = ceil(teacher_frames / 4); frame i covers audio [i*0.08, (i+1)*0.08) s
```

## Usage

Requires `funasr` and the frozen teacher backbone (Apache-2.0, from ModelScope/HF:
`iic/emotion2vec_plus_base`), then:

```python
from code.surgical_emotion import SurgicalEmotionEncoder

enc = SurgicalEmotionEncoder.from_pretrained(
    "/path/to/emotion2vec_plus_base",          # frozen teacher (funasr model dir)
    "surgical_emotion_module.pt",              # this repo's checkpoint
    device="cuda:0")
out = enc(wav_batch_16k, wav_lengths)
```

See `code/` for the full pipeline: module definition, self-distillation training,
evaluation, teacher-vs-student benchmark, and demo-gallery generation.

## Training

Self-distillation from the frozen teacher — **no emotion labels needed**:

- losses: frame cosine + LN-MSE (student upsampled to 50 Hz vs teacher states),
  utterance cosine, utterance KL, per-frame KL, intensity MSE
- data: RAVDESS speech, actors 1-18 (900 utterances); actors 19-24 held out
- 900 steps, AdamW lr 3e-4, fp16, single Tesla T4 (~5 min)

## Results (held-out actors 19-24)

| metric | value |
|---|---|
| top-1 agreement with teacher | **100 %** (56/56 files, all 8 emotions) |
| mean JS divergence vs teacher | 0.0010 |
| pooled-embedding cosine | 0.9994 |
| frame-level cosine (12.5->50 Hz reconstruction) | 0.9956 |
| rate contract | exact 12.5 Hz (+-1 frame boundary) |

Benchmark (T4, fp32): total student path 34.1 ms/utt (~142x realtime, RTF 0.0096);
surgical adapter overhead only 2.7 ms (~3.4 % params, ~2.3 % latency). Adapter
scales to 2395 utt/s at batch 8; fp16 autocast gives another ~22 %. Full numbers
in `reports/benchmark_report.json`.

`demo/` contains per-sample analysis plots (frame probabilities @12.5 Hz,
intensity vs target, teacher-vs-student bars) and numeric results for 7 samples
including the cross-lingual funasr Chinese test clip.

## Notes

- One-frame boundary tolerance at utterance end comes from the teacher's conv
  frontend (inherent to emotion2vec, not the surgery).
- The adapter was distilled on RAVDESS (CC BY-NC-SA 4.0): keep usage
  non-commercial or re-distill on permissively licensed speech (the recipe is
  label-free, so any emotional speech corpus works).
- Part of an S2ST project: next step injects `frame_embs` into dots.tts as a
  time-varying condition alongside its speaker x-vector path.