| --- |
| library_name: transformers.js |
| pipeline_tag: audio-classification |
| base_model: facebook/wav2vec2-base |
| datasets: |
| - myleslinder/crema-d |
| license: apache-2.0 |
| tags: |
| - audio |
| - speech-emotion-recognition |
| - onnx |
| - int8 |
| - transformers.js |
| metrics: |
| - accuracy |
| --- |
| |
| # Michi Audio Affect |
|
|
| An English speech-emotion classifier built for the fast listener in [Michi](https://github.com/cris-cmd/michi). It classifies 16 kHz mono speech into six coarse vocal-tone labels: |
|
|
| | Label | Meaning | |
| | --- | --- | |
| | `ang` | anger | |
| | `dis` | disgust | |
| | `fea` | fear | |
| | `hap` | happiness | |
| | `neu` | neutral | |
| | `sad` | sadness | |
|
|
| The repository contains the quantized ONNX runtime artifact. It does not contain training audio. |
|
|
| ## Intended use |
|
|
| This model is intended as a low-latency, secondary signal for conversational presentation. Michi combines its output with text emotion and deterministic rules; the model never changes plans, prices, reservations, or other application state. |
|
|
| Do not use it to diagnose mental health, infer stable personal traits, make high-impact decisions, or treat a predicted emotion as a fact about a person. |
|
|
| ## Training |
|
|
| - Base encoder: [`facebook/wav2vec2-base`](https://huggingface.co/facebook/wav2vec2-base), frozen at revision `0b5b8e8` |
| - Dataset: [`myleslinder/crema-d`](https://huggingface.co/datasets/myleslinder/crema-d), revision `8a11ae8` |
| - Input: 16 kHz speech audio |
| - Feature: mean-pooled hidden state from encoder layer 7 |
| - Head: multinomial logistic regression |
| - Split: actor-disjoint; eight actors held out for validation and the two local fixture actors excluded from training |
| - Export: `Wav2Vec2ForSequenceClassification` → ONNX → dynamic int8 quantization |
|
|
| The dependency- and revision-pinned build recipe is in `scripts/train-audio-affect-head.py` in the Michi repository. It is intended to be repeatable, but does not promise bit-for-bit identical artifacts across operating systems and hardware. |
|
|
| ## Validation |
|
|
| The logistic-regression head reached **78.2% accuracy** across six classes on an actor-disjoint validation split. The same validation split selected the best of 13 encoder layers, so this is a model-selection result rather than an estimate from an untouched test set. The score was calculated before ONNX export; the quantized artifact has runtime behavior checks, but has not been evaluated over the complete validation split. |
|
|
| | Class | Recall | |
| | --- | ---: | |
| | anger | 95.5% | |
| | disgust | 77.7% | |
| | fear | 63.4% | |
| | happiness | 73.2% | |
| | neutral | 93.8% | |
| | sadness | 67.9% | |
|
|
| The pre-export confusion matrix, validation actor IDs, and source revisions are in `TRAINING_REPORT.json`. |
|
|
| ## Usage with Transformers.js |
|
|
| ```javascript |
| import { pipeline } from "@huggingface/transformers"; |
| |
| const classify = await pipeline( |
| "audio-classification", |
| "cris-cmd/michi-audio-affect", |
| { dtype: "q8" }, |
| ); |
| |
| // Float32Array containing mono, 16 kHz PCM samples. |
| const scores = await classify(audio, { top_k: null }); |
| ``` |
|
|
| Michi pins a downloaded copy under `models/affect/michi-audio-affect` so it can run without a model-host request during a conversation. |
|
|
| ## Limitations |
|
|
| - Trained on acted English speech using a small set of fixed sentences. |
| - The six labels are intentionally coarse and do not cover natural emotional complexity. |
| - Accuracy will vary with language, accent, microphone quality, background noise, and spontaneous speech. |
| - Treat scores as uncertain presentation hints and combine them with text and explicit user intent. |
|
|
| ## Licenses and attribution |
|
|
| The exported model is released under Apache-2.0. The base encoder is Apache-2.0. CREMA-D is distributed under the Open Database License, with individual contents covered by the Database Contents License. Users are responsible for complying with the source dataset terms. |
|
|
| CREMA-D citation: |
|
|
| ```bibtex |
| @article{cao2014crema, |
| title={CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset}, |
| author={Cao, Houwei and Cooper, David G. and Keutmann, Michael K. and Gur, Ruben C. and Nenkova, Ani and Verma, Ragini}, |
| journal={IEEE Transactions on Affective Computing}, |
| volume={5}, |
| number={4}, |
| pages={377--390}, |
| year={2014}, |
| doi={10.1109/TAFFC.2014.2336244} |
| } |
| ``` |
|
|
