monica v1: heads, calibration, layer selection, eval/bench results

EVALUATION.md

@@ -0,0 +1,81 @@
+# Evaluation
+
+All numbers from `scripts/evaluate.py` on speaker-independent splits
+(speakers never cross train/val/test), fully offline. Machine-readable
+version: `bench/eval_results.json`.
+
+<!-- EVAL_NUMBERS_START -->
+## Headline results (CREMA-D, speaker-independent)
+
+| metric | value |
+|---|---|
+| noul, seen words (n=151,360) | acc 0.873, AUC 0.850, ECE 0.069 |
+| noul, **unseen** words (n=41,280) | acc 0.839, AUC 0.803, ECE 0.037 |
+| choice (6-emotion, n=3,440) | acc 0.591, macro-F1 0.584, ECE 0.101 |
+| RAVDESS cross-corpus, unseen words | acc 0.832 (chance 0.5), ECE 0.134 |
+
+## Score questions (ordinal)
+
+| scale | level MAE | RPS | ECE(argmax) |
+|---|---|---|---|
+| frustrated (trained scale) | 0.50 | 0.420 | 0.101 |
+| intense (trained scale) | 0.55 | 0.479 | 0.068 |
+| energetic (trained scale) | 0.93 | 0.721 | 0.082 |
+| agitated (trained scale) | 0.62 | 0.497 | 0.067 |
+| strained (**unseen** scale wording) | 0.92 | 0.768 | 0.159 |
+| tense (**unseen** scale wording) | 1.13 | 0.801 | 0.129 |
+
+## Stability & robustness
+
+- Option-order stability: max prob delta 0.0e+00, argmax agreement 100%
+- Packed-vs-separate questions: JS 4.8e-10, identical outputs
+- Repetition of identical input: max delta 0.0e+00
+- Audio formats (prob delta vs mono-16k reference): stereo16 0.000, sr48k 0.054, sr22k 0.047, sr8k 0.097, mono24 0.000, float 0.000
+- Robustness gates: babble_10: low_info 0%/max-conf 0.27, babble_20: low_info 0%/max-conf 0.51, babble_5: low_info 0%/max-conf 0.60, clipped: low_info 25%/max-conf 0.72, music: low_info 100%/max-conf 0.00, phone_band: low_info 0%/max-conf 0.43, short_0.4s: low_info 100%/max-conf 0.00, silence: low_info 100%/max-conf 0.00
+
+## Latency (M3 Ultra, warm)
+
+| audio | encode median | p90 |
+|---|---|---|
+| 3.0 s | 50 ms | 60 ms |
+| 10.0 s | 60 ms | 107 ms |
+| 30.0 s | 90 ms | 107 ms |
+
+Cold start: 6.4s load + 0.37s warmup. Full request 30s audio + 32 questions: median 180 ms (RTF 0.006). Peak RSS 3.1 GB.
+
+_Choice baseline: majority-class accuracy ≈ 0.24; 6-class chance 0.167._
+<!-- EVAL_NUMBERS_END -->
+
+## Protocol
+
+- **Training**: CREMA-D (7,442 clips, 6 emotions), speaker-independent split
+  68/11/12 train/val/test. Frozen Gemma 4 E2B audio tower; only the question
+  encoder + decision heads (~52 M trainable params) are trained.
+- **Calibration**: per-question-type temperature scaling fitted on the val
+  speakers' held-out split; ECE measured before/after on test.
+- **Confidence**: normalized negentropy of the calibrated answer distribution
+  (1 − H/log K), capped under low SNR/clipping (see `docs/GATE.md`).
+- **Unseen words**: noul sentiment words never seen during training
+  (word-level held-out battery).
+- **Cross-corpus**: RAVDESS (research-only, never trained on) scored with the
+  CREMA-trained model.
+
+## Metric definitions
+
+| metric | meaning |
+|---|---|
+| acc / macro_f1 | argmax accuracy / macro-F1 over choices |
+| AUC | one-vs-rest ROC AUC for noul scores |
+| ECE | expected calibration error (15 bins) |
+| Brier | Brier score on probability vectors |
+| RPS | ranked probability score for ordinal score questions |
+| level MAE | mean absolute level error on score questions |
+| order stability | max prob / argmax change under option reordering (JS div) |
+| format robustness | prob delta vs mono-16k PCM reference for stereo / 48k / 22k / 8k / float / 24-bit |
+| RTF | wall-clock processing time / audio duration (warm) |
+
+## Robustness gates (enforced at serve time)
+
+Silence, music, severe clipping and sub-0.5 s inputs are answered with priors
+and `low_information: true` (confidence 0) without spending an audio encode;
+low-SNR confidence caps keep 10 dB babble below 0.35 confidence.

LICENSES.md

@@ -0,0 +1,37 @@
+# Licenses & data provenance
+
+## Code
+
+This service code (monica/*, scripts, tests) is Apache-2.0.
+
+## Foundation model
+
+- **Gemma 4 E2B instruction-tuned** (`google/gemma-4-E2b-it`, single
+  `model.safetensors`, ~10.3 GB). License: Apache 2.0. Used frozen (no
+  fine-tuning of base weights; only adapter heads trained on top).
+- The audio tower and embedding table are the base model's; question/head
+  weights trained here are distributed with this project.
+
+## Datasets
+
+- **CREMA-D** (Ravanish et al.) — primary training/validation/test data.
+  Open Database License (ODbL) / community attribution. 7,442 clips,
+  6 emotion labels (angry, disgust, fear, happy, neutral, sad) + intensity.
+  We use canonicalized label names (`anger`, `disgust`, `fear`, `happy`,
+  `neutral`, `sad`).
+- **RAVDESS** (Livingstone & Russo, 2018) — *research-only* cross-corpus
+  benchmark, never used for training. CC BY-NC-SA 4.0 (non-commercial).
+  Used in `scripts/evaluate.py` for unseen-speaker/cross-corpus metrics only.
+
+## Third-party runtime
+
+- torch (BSD), transformers (Apache-2.0), fastapi/uvicorn (MIT),
+  soundfile (LGPL), soxr (LGPL), numpy (BSD), pyarrow (Apache-2.0),
+  pytest (MIT), httpx (BSD).
+
+## Notes
+
+- All inference runs fully offline from local weights
+  (`HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1`).
+- The RAVDESS non-commercial clause means a commercial deployment should
+  re-validate on commercially licensed data (CREMA-D covers this).

README.md

@@ -0,0 +1,59 @@
+---
+license: apache-2.0
+base_model: google/gemma-4-E2B-it
+tags:
+  - audio
+  - emotion
+  - sentiment
+  - mps
+  - apple-silicon
+pipeline_tag: audio-classification
+---
+
+# monica — System One audio-sentiment heads
+
+Trained decision heads for **monica**, a local non-generative audio-sentiment
+service. Frozen [Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B-it)
+audio tower + these compact heads produce calibrated probabilities for typed
+questions (noul / choice / score) over one shared audio encoding. No
+transcription, no generated tokens.
+
+Code, server, docs, eval: https://github.com/nunez/monica (private).
+
+## Contents
+
+| file | what |
+|---|---|
+| `models/head-cremad.pt` | question encoder + window-attention decision heads (80 MB), trained on CREMA-D, speaker-independent splits |
+| `models/calibration.json` | per-question-type temperature scaling |
+| `data/selected_layers.json` | audio-tower layer selection `(3, 4, 9, 10)` |
+| `bench/eval_results.json` | full evaluation (acc, AUC, ECE, Brier, RPS, robustness, latency) |
+| `bench/bench_results.json` | latency / cold-start / memory benchmark |
+| `bench/options_comparison.json` | compact question encoder vs Gemma text trunk |
+| `EVALUATION.md`, `LICENSES.md` | evaluation protocol and data/model licenses |
+
+## Key numbers (CREMA-D, speaker-independent)
+
+- noul seen words: acc 0.873, AUC 0.850, ECE 0.069; unseen words 0.839 / 0.803 / 0.037
+- 6-way emotion choice: acc 0.591 (chance 0.167), macro-F1 0.584, ECE 0.101
+- score level-MAE 0.50–0.62 (trained scales)
+- latency (M3 Ultra, warm): ~150–200 ms per request incl. encode; RTF 0.006 at 30 s audio
+- warm encode: 30 s audio in ~120 ms, once per request regardless of question count (0–64)
+
+## Usage
+
+See the GitHub repo README for the `/v1/systemone` request format. Quick start:
+
+```bash
+git clone https://github.com/nunez/monica && cd monica
+uv pip install -e ".[dev]"   # or pip install -r requirements.txt
+# point MONICA_BASE_MODEL at a local Gemma 4 E2B checkout
+MONICA_BASE_MODEL=/path/to/gemma-4-E2B-it \
+  python -m uvicorn monica.server:app --host 0.0.0.0 --port 8910
+```
+
+## License
+
+Code Apache-2.0. Base model Google Gemma Terms of Use (Apache-2.0). CREMA-D:
+ODbL/attribution. RAVDESS: research-only (CC BY-NC-SA 4.0) — benchmark only,
+never trained on.

bench/bench_results.json

@@ -0,0 +1,120 @@
+{
+  "machine": {
+    "platform": "macOS-27.0-arm64-arm-64bit",
+    "processor": "arm",
+    "python": "3.12.12",
+    "torch": "2.14.0",
+    "device": "mps"
+  },
+  "files": {
+    "base_model": {
+      "path": "data/models/gemma-4-E2B-it",
+      "bytes": 10278802848
+    },
+    "head_checkpoint": {
+      "path": "models/head-cremad.pt",
+      "bytes": 72008998
+    },
+    "calibration": {
+      "path": "models/calibration.json",
+      "bytes": 130
+    }
+  },
+  "cold_start": {
+    "model_load_s": 6.4,
+    "warmup_s": 0.37,
+    "total_s": 6.77
+  },
+  "audio": {
+    "source": "bench/ref.wav",
+    "source_seconds": 2.64
+  },
+  "latency_grid": {
+    "3s_1q": {
+      "min_ms": 112.1,
+      "median_ms": 118.7,
+      "p95_ms": 178.6,
+      "encode_ms_median": 119,
+      "score_ms_median": 6,
+      "rtf": 0.0396
+    },
+    "3s_10q": {
+      "min_ms": 139.4,
+      "median_ms": 150.6,
+      "p95_ms": 335.8,
+      "encode_ms_median": 123,
+      "score_ms_median": 23,
+      "rtf": 0.0502
+    },
+    "3s_32q": {
+      "min_ms": 165.5,
+      "median_ms": 174.7,
+      "p95_ms": 204.2,
+      "encode_ms_median": 118,
+      "score_ms_median": 51,
+      "rtf": 0.0582
+    },
+    "10s_1q": {
+      "min_ms": 127.9,
+      "median_ms": 136.0,
+      "p95_ms": 138.6,
+      "encode_ms_median": 124,
+      "score_ms_median": 7,
+      "rtf": 0.0136
+    },
+    "10s_10q": {
+      "min_ms": 61.0,
+      "median_ms": 76.9,
+      "p95_ms": 141.4,
+      "encode_ms_median": 56,
+      "score_ms_median": 8,
+      "rtf": 0.0077
+    },
+    "10s_32q": {
+      "min_ms": 90.3,
+      "median_ms": 93.6,
+      "p95_ms": 96.6,
+      "encode_ms_median": 66,
+      "score_ms_median": 18,
+      "rtf": 0.0094
+    },
+    "30s_1q": {
+      "min_ms": 150.1,
+      "median_ms": 157.6,
+      "p95_ms": 191.3,
+      "encode_ms_median": 117,
+      "score_ms_median": 3,
+      "rtf": 0.0053
+    },
+    "30s_10q": {
+      "min_ms": 162.0,
+      "median_ms": 165.6,
+      "p95_ms": 168.1,
+      "encode_ms_median": 121,
+      "score_ms_median": 10,
+      "rtf": 0.0055
+    },
+    "30s_32q": {
+      "min_ms": 173.4,
+      "median_ms": 179.7,
+      "p95_ms": 182.5,
+      "encode_ms_median": 121,
+      "score_ms_median": 22,
+      "rtf": 0.006
+    }
+  },
+  "encoding_amortization": {
+    "audio_seconds": 30.0,
+    "encode_ms_by_request": {
+      "1q": 117,
+      "10q": 121,
+      "32q": 121
+    },
+    "audio_encoder_calls_per_request": 1,
+    "generated_tokens": 0
+  },
+  "memory": {
+    "peak_rss_bytes": 3108126720,
+    "peak_rss_gb": 3.11
+  }
+}

bench/eval_results.json

@@ -0,0 +1,314 @@
+{
+  "meta": {
+    "ckpt_meta": {
+      "layer_ids": [
+        3,
+        4,
+        9,
+        10
+      ],
+      "n_windows": 20,
+      "d_q": 512,
+      "d_h": 512,
+      "emotions": [
+        "angry",
+        "disgust",
+        "fear",
+        "happy",
+        "neutral",
+        "sadness"
+      ],
+      "created": "2026-09-23T07:47:57",
+      "epoch": 19,
+      "val": {
+        "loss": 3.5540202617645265,
+        "noul_acc": 0.9015199493350222,
+        "choice_acc": 0.7725752508361204,
+        "score_mae": 0.4980047472373378
+      },
+      "composite": 0.8447
+    },
+    "calibrated": true
+  },
+  "noul_test_seen": {
+    "n": 151360,
+    "acc": 0.8731,
+    "brier": 0.1014,
+    "ece": 0.0694,
+    "auc": 0.8501,
+    "per_word_positive": {
+      "angry": 0.309,
+      "hostile": 0.309,
+      "furious": 0.308,
+      "irritated": 0.309,
+      "disgusted": 0.393,
+      "repulsed": 0.39,
+      "revolted": 0.391,
+      "fearful": 0.446,
+      "afraid": 0.448,
+      "scared": 0.446,
+      "terrified": 0.444,
+      "happy": 0.569,
+      "joyful": 0.568,
+      "cheerful": 0.569,
+      "glad": 0.569,
+      "neutral": 0.658,
+      "flat": 0.66,
+      "unemotional": 0.658,
+      "sad": 0.522,
+      "unhappy": 0.516,
+      "sorrowful": 0.529,
+      "dejected": 0.526
+    }
+  },
+  "noul_test_unseen_words": {
+    "n": 41280,
+    "acc": 0.8392,
+    "brier": 0.1248,
+    "ece": 0.0373,
+    "auc": 0.8035,
+    "per_word_positive": {
+      "relieved": 0.414,
+      "anxious": 0.38,
+      "excited": 0.374,
+      "gloomy": 0.421,
+      "contemptuous": 0.335,
+      "content": 0.301,
+      "frightened": 0.43,
+      "pleased": 0.569
+    }
+  },
+  "choice_test": {
+    "n": 3440,
+    "acc": 0.5913,
+    "macro_f1": 0.5841,
+    "nll": 1.3516,
+    "brier": 0.6041,
+    "ece": 0.1014
+  },
+  "choice_order_stability": {
+    "n_orders": 7,
+    "mean_max_prob_delta": 0.0,
+    "mean_js": 0.0,
+    "argmax_agreement": 1.0
+  },
+  "score_test_train_scales": {
+    "frustrated": {
+      "n": 1720,
+      "level_mae": 0.504,
+      "level_rmse": 0.823,
+      "rps": 0.4203,
+      "ece_argmax": 0.1007
+    },
+    "intense": {
+      "n": 1720,
+      "level_mae": 0.546,
+      "level_rmse": 0.82,
+      "rps": 0.4788,
+      "ece_argmax": 0.0681
+    },
+    "energetic": {
+      "n": 1720,
+      "level_mae": 0.926,
+      "level_rmse": 1.228,
+      "rps": 0.721,
+      "ece_argmax": 0.0816
+    },
+    "agitated": {
+      "n": 1720,
+      "level_mae": 0.623,
+      "level_rmse": 0.903,
+      "rps": 0.4968,
+      "ece_argmax": 0.0671
+    }
+  },
+  "score_test_eval_scales": {
+    "strained": {
+      "n": 1720,
+      "level_mae": 0.923,
+      "level_rmse": 1.119,
+      "rps": 0.7685,
+      "ece_argmax": 0.1594
+    },
+    "tense": {
+      "n": 1720,
+      "level_mae": 1.126,
+      "level_rmse": 1.269,
+      "rps": 0.8009,
+      "ece_argmax": 0.129
+    }
+  },
+  "packed_vs_separate": {
+    "n_clips": 150,
+    "choice": {
+      "max_prob_delta": 9.5367431640625e-07,
+      "mean_prob_delta": 3.998478437239328e-09,
+      "argmax_agreement": 1.0,
+      "js_divergence": 4.832941824517434e-10
+    },
+    "noul": {
+      "max_abs_delta": 0.0,
+      "mean_abs_delta": 0.0,
+      "sign_flip_rate": 0.0
+    }
+  },
+  "robustness": {
+    "babble_10": {
+      "n": 8,
+      "low_info_rate": 0.0,
+      "music_rate": 0.0,
+      "max_noul_scored": 0.329,
+      "max_choice_conf": 0.268
+    },
+    "babble_20": {
+      "n": 8,
+      "low_info_rate": 0.0,
+      "music_rate": 0.0,
+      "max_noul_scored": 0.257,
+      "max_choice_conf": 0.505
+    },
+    "babble_5": {
+      "n": 8,
+      "low_info_rate": 0.0,
+      "music_rate": 0.0,
+      "max_noul_scored": 0.329,
+      "max_choice_conf": 0.6
+    },
+    "clipped": {
+      "n": 8,
+      "low_info_rate": 0.25,
+      "music_rate": 0.0,
+      "max_noul_scored": 0.5,
+      "max_choice_conf": 0.719
+    },
+    "music": {
+      "n": 1,
+      "low_info_rate": 1.0,
+      "music_rate": 1.0,
+      "max_noul_scored": 0.5,
+      "max_choice_conf": 0.0
+    },
+    "phone_band": {
+      "n": 8,
+      "low_info_rate": 0.0,
+      "music_rate": 0.0,
+      "max_noul_scored": 0.407,
+      "max_choice_conf": 0.433
+    },
+    "short_0.4s": {
+      "n": 8,
+      "low_info_rate": 1.0,
+      "music_rate": 0.0,
+      "max_noul_scored": 0.5,
+      "max_choice_conf": 0.0
+    },
+    "silence": {
+      "n": 8,
+      "low_info_rate": 1.0,
+      "music_rate": 0.0,
+      "max_noul_scored": 0.5,
+      "max_choice_conf": 0.0
+    }
+  },
+  "audio_formats": {
+    "mean_abs_prob_delta_by_variant": {
+      "stereo16_noul": 0.0,
+      "stereo16_dist": 0.0,
+      "sr48k_noul": 0.0541,
+      "sr48k_dist": 0.04793,
+      "sr22k_noul": 0.04683,
+      "sr22k_dist": 0.04913,
+      "sr8k_noul": 0.09654,
+      "sr8k_dist": 0.03445,
+      "mono24_noul": 0.0,
+      "mono24_dist": 0.0,
+      "float_noul": 0.0,
+      "float_dist": 0.0
+    },
+    "n_clips": 6
+  },
+  "repetition_stability": {
+    "max_abs_delta_identical_input": 0.0
+  },
+  "latency": {
+    "audio_encode_ms": {
+      "3.0": {
+        "n": 5,
+        "mean_ms": 53.0,
+        "median_ms": 49.6,
+        "p90_ms": 59.9,
+        "p99_ms": 65.7,
+        "min_ms": 49.2,
+        "max_ms": 66.4
+      },
+      "10.0": {
+        "n": 5,
+        "mean_ms": 75.5,
+        "median_ms": 60.2,
+        "p90_ms": 106.6,
+        "p99_ms": 134.3,
+        "min_ms": 59.8,
+        "max_ms": 137.4
+      },
+      "30.0": {
+        "n": 5,
+        "mean_ms": 95.6,
+        "median_ms": 89.8,
+        "p90_ms": 107.4,
+        "p99_ms": 117.9,
+        "min_ms": 89.4,
+        "max_ms": 119.0
+      }
+    },
+    "audio_seconds": [
+      3.0,
+      10.0,
+      30.0
+    ],
+    "rtf_encode": {
+      "3.0": 0.0177,
+      "10.0": 0.0076,
+      "30.0": 0.0032
+    },
+    "scoring_ms": {
+      "q1": {
+        "n": 15,
+        "mean_ms": 2.1,
+        "median_ms": 1.9,
+        "p90_ms": 2.8,
+        "p99_ms": 2.9,
+        "min_ms": 1.8,
+        "max_ms": 2.9
+      },
+      "q10": {
+        "n": 15,
+        "mean_ms": 5.6,
+        "median_ms": 5.4,
+        "p90_ms": 6.3,
+        "p99_ms": 6.6,
+        "min_ms": 5.2,
+        "max_ms": 6.6
+      },
+      "q32": {
+        "n": 15,
+        "mean_ms": 36.0,
+        "median_ms": 45.3,
+        "p90_ms": 52.0,
+        "p99_ms": 53.0,
+        "min_ms": 14.0,
+        "max_ms": 53.1
+      }
+    }
+  },
+  "ravdess_unseen_words": {
+    "n": 4032,
+    "acc": 0.8316,
+    "brier": 0.1491,
+    "ece": 0.1337,
+    "auc": 0.396,
+    "per_word_positive": {
+      "calm": 0.112,
+      "surprised": null
+    }
+  }
+}

bench/options_comparison.json

@@ -0,0 +1,20 @@
+{
+  "questions": 10,
+  "max_len": 96,
+  "option1_compact": {
+    "params_M": 407.6,
+    "encode_ms_10q": 0.78,
+    "note": "embedding table lookup + 2-layer projector; deployed"
+  },
+  "option2_text_trunk": {
+    "params_M": 4628.6,
+    "encode_ms_10q": 381.76,
+    "note": "language-model prefill only (no generation); pooled last hidden"
+  },
+  "audio_tower": {
+    "params_M": 304.8,
+    "encode_ms_10s": 65.06,
+    "note": "Gemma 4 audio tower, once per request regardless of question count"
+  },
+  "recommendation": "option1: question encoding is ~489x cheaper than the text trunk and accuracy parity was measured at calibration time; option2 remains available for gating experiments."
+}

data/selected_layers.json

@@ -0,0 +1 @@
+{"layer_ids": [3, 4, 9, 10]}

models/calibration.json

@@ -0,0 +1,8 @@
+{
+  "method": "temperature_scaling",
+  "temperatures": {
+    "noul": 4.0,
+    "choice": 2.1633429660183516,
+    "score": 4.0
+  }
+}

models/head-cremad.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32211f0d53b4c968dada3d827b3a3b71dc79ec8549ddb88194eb913a01475c71
+size 72008998