# Evaluation

All numbers from `scripts/evaluate.py` on speaker-independent splits
(speakers never cross train/val/test), fully offline. Machine-readable
version: `bench/eval_results.json`.

<!-- EVAL_NUMBERS_START -->
## Headline results (CREMA-D, speaker-independent)

| metric | value |
|---|---|
| noul, seen words (n=151,360) | acc 0.873, AUC 0.850, ECE 0.069 |
| noul, **unseen** words (n=41,280) | acc 0.839, AUC 0.803, ECE 0.037 |
| choice (6-emotion, n=3,440) | acc 0.591, macro-F1 0.584, ECE 0.101 |
| RAVDESS cross-corpus, unseen words | acc 0.832 (chance 0.5), ECE 0.134 |

## Score questions (ordinal)

| scale | level MAE | RPS | ECE(argmax) |
|---|---|---|---|
| frustrated (trained scale) | 0.50 | 0.420 | 0.101 |
| intense (trained scale) | 0.55 | 0.479 | 0.068 |
| energetic (trained scale) | 0.93 | 0.721 | 0.082 |
| agitated (trained scale) | 0.62 | 0.497 | 0.067 |
| strained (**unseen** scale wording) | 0.92 | 0.768 | 0.159 |
| tense (**unseen** scale wording) | 1.13 | 0.801 | 0.129 |

## Stability & robustness

- Option-order stability: max prob delta 0.0e+00, argmax agreement 100%
- Packed-vs-separate questions: JS 4.8e-10, identical outputs
- Repetition of identical input: max delta 0.0e+00
- Audio formats (prob delta vs mono-16k reference): stereo16 0.000, sr48k 0.054, sr22k 0.047, sr8k 0.097, mono24 0.000, float 0.000
- Robustness gates: babble_10: low_info 0%/max-conf 0.27, babble_20: low_info 0%/max-conf 0.51, babble_5: low_info 0%/max-conf 0.60, clipped: low_info 25%/max-conf 0.72, music: low_info 100%/max-conf 0.00, phone_band: low_info 0%/max-conf 0.43, short_0.4s: low_info 100%/max-conf 0.00, silence: low_info 100%/max-conf 0.00

## Latency (M3 Ultra, warm)

| audio | encode median | p90 |
|---|---|---|
| 3.0 s | 50 ms | 60 ms |
| 10.0 s | 60 ms | 107 ms |
| 30.0 s | 90 ms | 107 ms |

Cold start: 6.4s load + 0.37s warmup. Full request 30s audio + 32 questions: median 180 ms (RTF 0.006). Peak RSS 3.1 GB.

_Choice baseline: majority-class accuracy ≈ 0.24; 6-class chance 0.167._
<!-- EVAL_NUMBERS_END -->

## Protocol

- **Training**: CREMA-D (7,442 clips, 6 emotions), speaker-independent split
  68/11/12 train/val/test. Frozen Gemma 4 E2B audio tower; only the question
  encoder + decision heads (~52 M trainable params) are trained.
- **Calibration**: per-question-type temperature scaling fitted on the val
  speakers' held-out split; ECE measured before/after on test.
- **Confidence**: normalized negentropy of the calibrated answer distribution
  (1 − H/log K), capped under low SNR/clipping (see `docs/GATE.md`).
- **Unseen words**: noul sentiment words never seen during training
  (word-level held-out battery).
- **Cross-corpus**: RAVDESS (research-only, never trained on) scored with the
  CREMA-trained model.

## Metric definitions

| metric | meaning |
|---|---|
| acc / macro_f1 | argmax accuracy / macro-F1 over choices |
| AUC | one-vs-rest ROC AUC for noul scores |
| ECE | expected calibration error (15 bins) |
| Brier | Brier score on probability vectors |
| RPS | ranked probability score for ordinal score questions |
| level MAE | mean absolute level error on score questions |
| order stability | max prob / argmax change under option reordering (JS div) |
| format robustness | prob delta vs mono-16k PCM reference for stereo / 48k / 22k / 8k / float / 24-bit |
| RTF | wall-clock processing time / audio duration (warm) |

## Robustness gates (enforced at serve time)

Silence, music, severe clipping and sub-0.5 s inputs are answered with priors
and `low_information: true` (confidence 0) without spending an audio encode;
low-SNR confidence caps keep 10 dB babble below 0.35 confidence.