| --- |
| license: apache-2.0 |
| tags: |
| - speech-emotion-recognition |
| - emotion2vec |
| - speech-to-speech-translation |
| - tts |
| - conditioning |
| library_name: funasr |
| --- |
| |
| # Surgical Emotion Module — emotion2vec_plus_base @ 12.5 Hz |
|
|
| A surgically modified **emotion2vec_plus_base** that outputs **continuous, frame-level |
| emotion vectors at 12.5 Hz** (80 ms/frame) instead of a single pooled utterance |
| vector — designed as the emotion-conditioning front-end of a Speech-to-Speech |
| Translation (S2ST) system built on top of **dots.tts** (its LLM token rate is |
| 6.25 Hz, so exactly **2 emotion frames align to every LLM token**). |
|
|
| ## What the surgery changes |
|
|
| | | stock emotion2vec_plus_base | this module | |
| |---|---|---| |
| | output | 1 pooled 768-d utterance vector | per-frame **768-d continuous vectors @ 12.5 Hz** | |
| | emotion probs | utterance-level, 9 classes | **per-frame** 9 classes + utterance-level | |
| | intensity | none | **continuous [0,1] per frame** | |
| | dynamics | discarded by pooling | preserved (within-word intensity changes) | |
|
|
| Architecture of the implant (3.18 M trainable params, the 93.8 M backbone stays frozen): |
|
|
| ``` |
| frozen emotion2vec_plus_base backbone (50 Hz frame states, 768-d) |
| -> LayerNorm -> Linear(768->512) |
| -> 3x depthwise-separable conv fusion (k=9, dilation 1/2/4) |
| -> grouped soft-attention pooling over G=4 frames (50 Hz -> 12.5 Hz) |
| -> LayerNorm -> Linear(512->768) (continuous emotion vectors) |
| -> heads: emotion logits (9) + intensity (sigmoid) |
| ``` |
|
|
| Intensity is distilled from a confidence-gated expression-energy target |
| (sqrt(max-prob) x utterance-normalized RMS envelope), which gives real temporal |
| dynamics instead of the saturated confidence proxy. |
|
|
| ## Outputs |
|
|
| ```python |
| out = enc(wav_16k, wav_lengths) |
| out["frame_embs"] # (B, T, 768) continuous emotion vectors @ 12.5 Hz |
| out["frame_probs"] # (B, T, 9) per-frame emotion probabilities |
| out["frame_intensity"] # (B, T, 1) per-frame intensity in [0,1] |
| out["utt_emb"] # (B, 768) pooled utterance embedding |
| out["utt_probs"] # (B, 9) |
| out["frame_lengths"] # (B,) |
| # T = ceil(teacher_frames / 4); frame i covers audio [i*0.08, (i+1)*0.08) s |
| ``` |
|
|
| ## Usage |
|
|
| Requires `funasr` and the frozen teacher backbone (Apache-2.0, from ModelScope/HF: |
| `iic/emotion2vec_plus_base`), then: |
|
|
| ```python |
| from code.surgical_emotion import SurgicalEmotionEncoder |
| |
| enc = SurgicalEmotionEncoder.from_pretrained( |
| "/path/to/emotion2vec_plus_base", # frozen teacher (funasr model dir) |
| "surgical_emotion_module.pt", # this repo's checkpoint |
| device="cuda:0") |
| out = enc(wav_batch_16k, wav_lengths) |
| ``` |
|
|
| See `code/` for the full pipeline: module definition, self-distillation training, |
| evaluation, teacher-vs-student benchmark, and demo-gallery generation. |
|
|
| ## Training |
|
|
| Self-distillation from the frozen teacher — **no emotion labels needed**: |
|
|
| - losses: frame cosine + LN-MSE (student upsampled to 50 Hz vs teacher states), |
| utterance cosine, utterance KL, per-frame KL, intensity MSE |
| - data: RAVDESS speech, actors 1-18 (900 utterances); actors 19-24 held out |
| - 900 steps, AdamW lr 3e-4, fp16, single Tesla T4 (~5 min) |
|
|
| ## Results (held-out actors 19-24) |
|
|
| | metric | value | |
| |---|---| |
| | top-1 agreement with teacher | **100 %** (56/56 files, all 8 emotions) | |
| | mean JS divergence vs teacher | 0.0010 | |
| | pooled-embedding cosine | 0.9994 | |
| | frame-level cosine (12.5->50 Hz reconstruction) | 0.9956 | |
| | rate contract | exact 12.5 Hz (+-1 frame boundary) | |
|
|
| Benchmark (T4, fp32): total student path 34.1 ms/utt (~142x realtime, RTF 0.0096); |
| surgical adapter overhead only 2.7 ms (~3.4 % params, ~2.3 % latency). Adapter |
| scales to 2395 utt/s at batch 8; fp16 autocast gives another ~22 %. Full numbers |
| in `reports/benchmark_report.json`. |
|
|
| `demo/` contains per-sample analysis plots (frame probabilities @12.5 Hz, |
| intensity vs target, teacher-vs-student bars) and numeric results for 7 samples |
| including the cross-lingual funasr Chinese test clip. |
|
|
| ## Notes |
|
|
| - One-frame boundary tolerance at utterance end comes from the teacher's conv |
| frontend (inherent to emotion2vec, not the surgery). |
| - The adapter was distilled on RAVDESS (CC BY-NC-SA 4.0): keep usage |
| non-commercial or re-distill on permissively licensed speech (the recipe is |
| label-free, so any emotional speech corpus works). |
| - Part of an S2ST project: next step injects `frame_embs` into dots.tts as a |
| time-varying condition alongside its speaker x-vector path. |
|
|
