amir0907/student-v4

Surgical Emotion Module v4 — emotion2vec_plus_base @ 6.25 Hz (1 frame per dots.tts token)

Fourth generation of the surgical emotion module (v1: amir0907/student · v2: amir0907/student-v2 · v3: amir0907/student-v3). Same frozen backbone and adapter architecture, two interface/recipe changes:

1. Frame rate halved to 6.25 Hz (group_size 4 → 8): exactly 1 emotion frame per dots.tts LLM token (48000/7680 = 6.25 Hz, 160 ms) — zero interpolation glue at injection time; the soft-attention pooling fuses the full 160 ms window.
2. Intensity target + losses redesigned to fix v3's collapsed intensity (v3 output a near-constant ≈0.31 regardless of emotion — see diagnostics below).

v4 intensity recipe

target(t) = p_emo(t) * clip(rms_raw(t) / GLOBAL_REF, 0, 1)

p_emo(t)    max teacher prob over the 6 EMOTIONAL classes (angry, disgusted,
            fearful, happy, sad, surprised) — neutral/other/unk EXCLUDED and NOT
            sqrt-ed (v3's sqrt(max-prob incl. neutral) saturated at ~0.99, so the
            "conf-gated" target degenerated to a per-utterance min-max loudness
            envelope carrying no emotion information)
rms_raw     raw 50 Hz RMS envelope, normalized by a GLOBAL reference level
            (median over the pool of per-utterance P90 RMS) so cross-utterance
            loudness differences survive (v3 min-max erased them)
losses      MSE + Pearson-correlation + log-std matching (v3's plain MSE to a
            noisy target rewarded predicting the mean → mean-collapse)

Why v3's intensity was dead (benchmarks that motivated v4)

On 32 held-out RAVDESS files: v3 gate sqrt(maxprob) = 0.992 ± 0.018 mean, 95.8 % of frames > 0.9 confidence; target std 0.290 ≈ plain-RMS std 0.291; student std 0.022 (7 % of target variance); corr to gated target 0.390 ≈ corr to plain RMS 0.388; neutral vs angry utterance-mean intensity 0.324 vs 0.335.

Results — identical held-out inputs (RAVDESS actors 19-24 + zh clip, vs frozen teacher)

metric	v1	v2	v4
top-1 agreement	100 %	100 %	100 %
mean JS divergence ↓	0.00035	0.00022	0.00044
pooled-embedding cosine	0.99935	0.99955	0.99926
frame-level cosine (up to 50 Hz)	0.99524	0.99572	0.99198
intensity corr (new target)	0.237	0.239	0.594
intensity curve std (dynamics)	0.030	0.035	0.183
normal(01) vs strong(02) mean-intensity diff	+0.021	+0.027	+0.047
intensity Cohen-d (336 emotional files)	0.57	0.66	0.26

Rate contract exact at 6.25 Hz for 1/2.5/5/10 s inputs. Authoritative numbers: reports/v1_v2_v4_compare.json and reports/eval_report_v4.json. On the fidelity subset (60 files) v4 reaches mean int-corr 0.563 vs the same metric for v3 (0.288 on its own proxy).

Ceiling note: the target itself separates RAVDESS 01/02 with Cohen-d 0.23 (v4 student: 0.26) — v4 is at the ceiling of what this label-free proxy encodes. Pushing separation further needs a signal that actually encodes acted intensity (e.g. absolute-loudness calibration or weak supervision from RAVDESS/CREMA-D intensity labels).

Training

Self-distillation from the frozen teacher — no labels used; same multi-corpus pool as v3 (RAVDESS speech+song actors 1-18 + CREMA-D + TESS + ESD en+zh, 50,618 utterances; 4-shard cache precomputed on 2×T4). 4000 AdamW steps, batch 8, fp16, single T4 (~14 min). Held-out gate: RAVDESS actors 19-24 (never trained on).

Demo (demo/)

9 samples: one strong-intensity utterance per emotion (03-08) + neutral + the funasr zh clip — each with the actual test audio, the v4 analysis plot (frame probabilities @6.25 Hz, intensity vs target, teacher-vs-v4 bars) and numeric results. RAVDESS audio is CC BY-NC-SA 4.0 (Livingstone & Russo 2018) — non-commercial use only; the zh clip is MIT.

Usage

from code.surgical_emotion import SurgicalEmotionEncoder

enc = SurgicalEmotionEncoder.from_pretrained(
    "/path/to/emotion2vec_plus_base",   # frozen teacher (funasr dir, Apache-2.0)
    "surgical_emotion_module.pt",       # this repo's checkpoint (G=8 is restored automatically)
    device="cuda:0")
out = enc(wav_16k, wav_lengths)
# frame_embs (B,T,768) @6.25Hz | frame_probs (B,T,9) | frame_intensity (B,T,1)
# utt_emb (B,768) | utt_probs (B,9) | frame_lengths (B,)
# T = ceil(duration_s * 6.25); frame i covers audio [i*0.16, (i+1)*0.16) s

Notes

• Adapter identical to v1-v3 (3.18 M params, ~2.7 ms overhead); rate change is a config switch (group_size 8), so a 12.5 Hz variant remains reproducible.
• Distilled on RAVDESS (CC BY-NC-SA) + CREMA-D/TESS/ESD: keep non-commercial or re-distill on permissive speech (label-free recipe, any corpus works).
• Part of an S2ST project: frame_embs are designed for per-token time-varying conditioning of dots.tts (1 emotion frame per 6.25 Hz LLM token).