Evaluation
All numbers from scripts/evaluate.py on speaker-independent splits
(speakers never cross train/val/test), fully offline. Machine-readable
version: bench/eval_results.json.
Headline results (CREMA-D, speaker-independent)
| metric | value |
|---|---|
| noul, seen words (n=151,360) | acc 0.873, AUC 0.850, ECE 0.069 |
| noul, unseen words (n=41,280) | acc 0.839, AUC 0.803, ECE 0.037 |
| choice (6-emotion, n=3,440) | acc 0.591, macro-F1 0.584, ECE 0.101 |
| RAVDESS cross-corpus, unseen words | acc 0.832 (chance 0.5), ECE 0.134 |
Score questions (ordinal)
| scale | level MAE | RPS | ECE(argmax) |
|---|---|---|---|
| frustrated (trained scale) | 0.50 | 0.420 | 0.101 |
| intense (trained scale) | 0.55 | 0.479 | 0.068 |
| energetic (trained scale) | 0.93 | 0.721 | 0.082 |
| agitated (trained scale) | 0.62 | 0.497 | 0.067 |
| strained (unseen scale wording) | 0.92 | 0.768 | 0.159 |
| tense (unseen scale wording) | 1.13 | 0.801 | 0.129 |
Stability & robustness
- Option-order stability: max prob delta 0.0e+00, argmax agreement 100%
- Packed-vs-separate questions: JS 4.8e-10, identical outputs
- Repetition of identical input: max delta 0.0e+00
- Audio formats (prob delta vs mono-16k reference): stereo16 0.000, sr48k 0.054, sr22k 0.047, sr8k 0.097, mono24 0.000, float 0.000
- Robustness gates: babble_10: low_info 0%/max-conf 0.27, babble_20: low_info 0%/max-conf 0.51, babble_5: low_info 0%/max-conf 0.60, clipped: low_info 25%/max-conf 0.72, music: low_info 100%/max-conf 0.00, phone_band: low_info 0%/max-conf 0.43, short_0.4s: low_info 100%/max-conf 0.00, silence: low_info 100%/max-conf 0.00
Latency (M3 Ultra, warm)
| audio | encode median | p90 |
|---|---|---|
| 3.0 s | 50 ms | 60 ms |
| 10.0 s | 60 ms | 107 ms |
| 30.0 s | 90 ms | 107 ms |
Cold start: 6.4s load + 0.37s warmup. Full request 30s audio + 32 questions: median 180 ms (RTF 0.006). Peak RSS 3.1 GB.
Choice baseline: majority-class accuracy ≈ 0.24; 6-class chance 0.167.
Protocol
- Training: CREMA-D (7,442 clips, 6 emotions), speaker-independent split 68/11/12 train/val/test. Frozen Gemma 4 E2B audio tower; only the question encoder + decision heads (~52 M trainable params) are trained.
- Calibration: per-question-type temperature scaling fitted on the val speakers' held-out split; ECE measured before/after on test.
- Confidence: normalized negentropy of the calibrated answer distribution
(1 − H/log K), capped under low SNR/clipping (see
docs/GATE.md). - Unseen words: noul sentiment words never seen during training (word-level held-out battery).
- Cross-corpus: RAVDESS (research-only, never trained on) scored with the CREMA-trained model.
Metric definitions
| metric | meaning |
|---|---|
| acc / macro_f1 | argmax accuracy / macro-F1 over choices |
| AUC | one-vs-rest ROC AUC for noul scores |
| ECE | expected calibration error (15 bins) |
| Brier | Brier score on probability vectors |
| RPS | ranked probability score for ordinal score questions |
| level MAE | mean absolute level error on score questions |
| order stability | max prob / argmax change under option reordering (JS div) |
| format robustness | prob delta vs mono-16k PCM reference for stereo / 48k / 22k / 8k / float / 24-bit |
| RTF | wall-clock processing time / audio duration (warm) |
Robustness gates (enforced at serve time)
Silence, music, severe clipping and sub-0.5 s inputs are answered with priors
and low_information: true (confidence 0) without spending an audio encode;
low-SNR confidence caps keep 10 dB babble below 0.35 confidence.