| --- |
| license: bsd-2-clause |
| language: en |
| library_name: onnx |
| pipeline_tag: audio-classification |
| base_model: laion/whisper-tiny-base-emotion-voice-burst |
| tags: |
| - pipecat |
| - speech-emotion-recognition |
| - sarcasm |
| - prosody |
| - onnx |
| --- |
| |
| # Pipecat Voice Reader (rc1) |
|
|
| > **β οΈ Experimental.** This is a first release candidate, published for trying out and feedback. Its outputs, |
| > names and calibration may change between releases, and its readings are cues for an LLM to weigh, not verdicts |
| > to act on alone. Pin a revision, and read the limitations below before relying on it. |
|
|
| A small CPU model that hears how a user *sounds*: tone of voice, sarcasm in the delivery, energy, laughter and |
| speech rate. It is the voice half of a pair: it is made to run beside Pipecat's |
| [**Tone Reader**](https://huggingface.co/pipecat-ai/tone-models) (`PipecatToneReader`), which reads the *words*. |
| It reads the last 8 s of each user turn, once, when the user stops speaking, and Pipecat's `VoiceReaderProcessor` |
| writes the reading as RCS perception at the head of the user's message, after the Tone Reader's: |
|
|
| ``` |
| ([emotion happy=0.62, neutral=0.30],[sarcasm value=0.08],[tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great. |
| ``` |
|
|
| ## Made to pair with the Tone Reader |
|
|
| The two readers split the work, and each declares its own signals so the LLM sees both side by side: |
|
|
| | | [Tone Reader](https://huggingface.co/pipecat-ai/tone-models) (the words) | Voice Reader (the voice) | |
| |---|---|---| |
| | Reads | the final transcript, with the bot's previous line | the last 8 s of the user's audio | |
| | Writes | `emotion` (nine labels), `sarcasm` | `tone` (four labels), `sarcasm_voice`, `arousal`, `laughter`, `speech_rate` | |
| | Strong on | what the words say: emotion, and sarcasm the words give away | what the words don't: a flat or frustrated delivery, sadness behind neutral words, sarcasm said with sincere words | |
|
|
| The voice reader never writes `emotion` or `sarcasm`. Its `sarcasm_voice` is a weak cue on its own, and its |
| declared description tells the LLM to weigh it with the Tone Reader's `sarcasm`: the case it earns its place in is |
| words that read sincere, said in a voice that isn't. For example, from our test clips, "He's the greatest, I'd |
| say." said sarcastically reads as happy and 2 % sarcastic from the words, and 15 % (above the 6.8 % threshold) |
| from the voice. |
|
|
| It does run without the Tone Reader: the LLM then gets `tone` instead of `emotion`, and no `sarcasm` from the |
| words to weigh `sarcasm_voice` against, so we recommend running both. |
|
|
| ## Variants |
|
|
| This release carries two sizes of the same model, one folder each, read by the same runtime: |
|
|
| | Folder | `variant` | Params | int8 ONNX | Read time, p50 (M1 Max, 4 / 2 / 1 threads) | 1 thread, x86 server | Choose it for | |
| |---|---|---|---|---|---|---| |
| | `voice-reader-base/` | `"base"` (default) | 26M | 28.6 MB | 33 / 56 / 100 ms | ~100β150 ms | the best tone and sadness reading; adds the `sad` output | |
| | `voice-reader-tiny/` | `"tiny"` | 12.5M | 13.3 MB | 16 / 26 / 47 ms | ~45 ms | one-vCPU deployments: sarcasm about as well, half the CPU | |
|
|
| The read starts when the VAD says the user stopped and runs while the speech-to-text finalises, so it rarely |
| delays the transcription; the cost to watch is CPU at the end of each turn, beside the VAD, the turn detector and |
| the reply. Give `base` two threads where the machine allows. |
|
|
| Each folder has `model.int8.onnx` (the default), `model.onnx` (fp32), `config.json` (calibration, threshold, |
| output names) and `MANIFEST.json` (SHA-256 of every file, checked by the runtime on download). |
|
|
| ## Use |
|
|
| With the Tone Reader (`pipecat-ai[tone]`) before it, as intended: |
|
|
| ```python |
| from pipecat.processors.tone.reader import PipecatToneReader |
| from pipecat_voice.reader import VoiceReaderParams, VoiceReaderProcessor |
| |
| pipeline = Pipeline([ |
| transport.input(), |
| stt, |
| PipecatToneReader(), # the words: emotion, sarcasm |
| VoiceReaderProcessor(params=VoiceReaderParams(revision="rc1", variant="base", threads=2)), # the voice |
| user_aggregator, |
| llm, |
| tts, |
| transport.output(), |
| ]) |
| ``` |
|
|
| It goes after the STT (it needs the transcriptions, and the STT passes the audio on) and after the Tone Reader, |
| whose signal declarations it merges with its own, so an LLM that keeps only the latest declaration still sees |
| both. The runtime is numpy and onnxruntime; the model downloads from |
| this repository at the pinned revision the first time. |
|
|
| ## Outputs |
|
|
| | Signal | Published to the LLM | Meaning | |
| |---|---|---| |
| | `tone` | yes, top two | how the voice sounds: positive, neutral, sad, frustrated (temperature-calibrated) | |
| | `sarcasm_voice` | yes | the chance the delivery alone sounds sarcastic, Platt-calibrated on real conversation; below the release threshold (0.068) it is published under 0.5, as a guess; dropped under laughter | |
| | `arousal` | yes | vocal energy, 0β1 | |
| | `laughter` | yes | 0β1 | |
| | `speech_rate` | yes | syllables per second | |
| | `upset` | no, raw reading only | how clearly the speaker is upset, 0β1 | |
| | `sad` (`base` only) | no, raw reading only | how likely listeners would hear sadness, 0β1 | |
|
|
| Nothing is written for a turn with under 0.8 s of speech. The raw reading, including `upset` and `sad`, reaches |
| the application through the processor's `on_reading` event. |
|
|
| ## Evaluation |
|
|
| Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5) hearing |
| each clip; a turn counts as sarcastic only when both say so (judge agreement on sarcasm: kappa 0.42). Test |
| speakers never appear in training. |
|
|
| | | `base` | `tiny` | |
| |---|---|---| |
| | **Sarcasm**, everyday conversation (4,366 turns, 203 sarcastic; Open Yap, otoSpeech, HarperValley), AUROC | 0.917 | 0.904 | |
| | β at the release threshold: F1 / recall / false alarms on sincere turns | 0.575 / 0.50 / 1.3 % | 0.560 / 0.48 / 1.3 % | |
| | β mean AUROC over four test sets (two conversational, Emolia, MUStARD++) | 0.791 | 0.788 | |
| | β MUStARD++ (TV, audio only, evaluation only), AUROC | 0.656 | 0.654 | |
| | **Tone** macro-F1: conversation / Emolia / CREMA-D (acted) / EmoNet-Voice (synthetic) | 0.706 / 0.661 / 0.887 / 0.563 | 0.712 / 0.627 / 0.872 / 0.457 | |
| | **Sadness**: tone recall for sad, mean of four sets | 0.53 | 0.38 | |
| | β `sad` output, AUROC: conversation (any judge heard sadness) / Emolia | 0.762 / 0.929 | β | |
| | **Upset**, mean AUROC (three sets) | 0.867 | 0.856 | |
| | **Arousal**, CCC against the training target | 0.89 | 0.88 | |
| | **Laughter**, AUROC / F1 | 0.994 / 0.72 | 0.991 / 0.70 | |
| | int8 vs fp32: tone agreement / sarcasm p95 \|Ξp\| | 0.983 / 0.011 | 0.977 / 0.010 | |
|
|
| ## Limitations |
|
|
| - **Sarcasm from the voice alone is a weak cue.** At the threshold it catches about half of the sarcasm both |
| judges heard, at about 1 false alarm in 75 sincere turns. Weigh it with the words, never alone. On synthetic |
| (TTS) sarcasm it is near chance (AUROC 0.60 on our 20 demo pairs). |
| - **Sadness in conversation is rare** (2β6 % of turns), so even a good detector's per-turn flags are mostly |
| wrong (12β25 % precision at a 5 % false-alarm rate). Treat `sad` as a graded cue or average it over a |
| conversation. It is not a measure of depression or of anyone's mental health, and a single turn cannot show |
| either. |
| - **Acted, theatrical sadness** is under-read: 23 of 34 CREMA-D sad clips. |
| - English only; trained on conversational and podcast speech, 8 kHz telephone audio included. |
| - Tone labels are LLM judgments of the audio, not human ratings. |
|
|
| ## Training |
|
|
| - **Encoder**: Whisper encoder initialised from LAION's affect-tuned |
| [`laion/whisper-tiny-base-emotion-voice-burst`](https://huggingface.co/laion/whisper-tiny-base-emotion-voice-burst) |
| (CC BY 4.0), base and tiny encoders respectively, cut to an 8 s window, with masked statistics pooling and one |
| small head per output. |
| - **Data** (about 119k training windows): Open Yap 1K and otoSpeech conversations, HarperValleyBank calls |
| (CC BY 4.0), the English YODAS part of Emilia from `laion/Emolia` (CC BY 4.0), and CREMA-D (ODbL / DbCL). |
| Augmented with gain, noise, telephone band-limiting and codecs; never speed or pitch, since rate and pitch are |
| the signal. |
| - **Labels**: the two judges' verdicts on sarcasm, tone and upset; where they disagreed on tone, the mean of two |
| audio emotion models (emotion2vec+ large and the LAION model) at half weight; the `sad` head on the share of |
| judges who heard sadness. |
| - **Tiny**: distilled from the base model's predictions at half weight, with base-labelled Emolia clips added. |
| - **Calibration** on real conversational dev turns: temperature for tone, Platt for sarcasm and laughter; the |
| sarcasm threshold keeps false alarms at 2 % of sincere dev turns. |
|
|
| Evaluation-only data, never trained on: MUStARD++ (CC BY-NC) and EmoNet-Voice Bench's clearly rated clips. |
|
|
