---
library_name: transformers.js
pipeline_tag: audio-classification
base_model: facebook/wav2vec2-base
datasets:
  - myleslinder/crema-d
license: apache-2.0
tags:
  - audio
  - speech-emotion-recognition
  - onnx
  - int8
  - transformers.js
metrics:
  - accuracy
---

# Michi Audio Affect

An English speech-emotion classifier built for the fast listener in [Michi](https://github.com/cris-cmd/michi). It classifies 16 kHz mono speech into six coarse vocal-tone labels:

| Label | Meaning |
| --- | --- |
| `ang` | anger |
| `dis` | disgust |
| `fea` | fear |
| `hap` | happiness |
| `neu` | neutral |
| `sad` | sadness |

The repository contains the quantized ONNX runtime artifact. It does not contain training audio.

## Intended use

This model is intended as a low-latency, secondary signal for conversational presentation. Michi combines its output with text emotion and deterministic rules; the model never changes plans, prices, reservations, or other application state.

Do not use it to diagnose mental health, infer stable personal traits, make high-impact decisions, or treat a predicted emotion as a fact about a person.

## Training

- Base encoder: [`facebook/wav2vec2-base`](https://huggingface.co/facebook/wav2vec2-base), frozen at revision `0b5b8e8`
- Dataset: [`myleslinder/crema-d`](https://huggingface.co/datasets/myleslinder/crema-d), revision `8a11ae8`
- Input: 16 kHz speech audio
- Feature: mean-pooled hidden state from encoder layer 7
- Head: multinomial logistic regression
- Split: actor-disjoint; eight actors held out for validation and the two local fixture actors excluded from training
- Export: `Wav2Vec2ForSequenceClassification` → ONNX → dynamic int8 quantization

The dependency- and revision-pinned build recipe is in `scripts/train-audio-affect-head.py` in the Michi repository. It is intended to be repeatable, but does not promise bit-for-bit identical artifacts across operating systems and hardware.

## Validation

The logistic-regression head reached **78.2% accuracy** across six classes on an actor-disjoint validation split. The same validation split selected the best of 13 encoder layers, so this is a model-selection result rather than an estimate from an untouched test set. The score was calculated before ONNX export; the quantized artifact has runtime behavior checks, but has not been evaluated over the complete validation split.

| Class | Recall |
| --- | ---: |
| anger | 95.5% |
| disgust | 77.7% |
| fear | 63.4% |
| happiness | 73.2% |
| neutral | 93.8% |
| sadness | 67.9% |

The pre-export confusion matrix, validation actor IDs, and source revisions are in `TRAINING_REPORT.json`.

## Usage with Transformers.js

```javascript
import { pipeline } from "@huggingface/transformers";

const classify = await pipeline(
  "audio-classification",
  "cris-cmd/michi-audio-affect",
  { dtype: "q8" },
);

// Float32Array containing mono, 16 kHz PCM samples.
const scores = await classify(audio, { top_k: null });
```

Michi pins a downloaded copy under `models/affect/michi-audio-affect` so it can run without a model-host request during a conversation.

## Limitations

- Trained on acted English speech using a small set of fixed sentences.
- The six labels are intentionally coarse and do not cover natural emotional complexity.
- Accuracy will vary with language, accent, microphone quality, background noise, and spontaneous speech.
- Treat scores as uncertain presentation hints and combine them with text and explicit user intent.

## Licenses and attribution

The exported model is released under Apache-2.0. The base encoder is Apache-2.0. CREMA-D is distributed under the Open Database License, with individual contents covered by the Database Contents License. Users are responsible for complying with the source dataset terms.

CREMA-D citation:

```bibtex
@article{cao2014crema,
  title={CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset},
  author={Cao, Houwei and Cooper, David G. and Keutmann, Michael K. and Gur, Ruben C. and Nenkova, Ani and Verma, Ragini},
  journal={IEEE Transactions on Affective Computing},
  volume={5},
  number={4},
  pages={377--390},
  year={2014},
  doi={10.1109/TAFFC.2014.2336244}
}
```