Add WakeHuBERT wake-word models, models.json and model card

README.md

@@ -0,0 +1,152 @@
+---
+license: apache-2.0
+library_name: onnx
+language:
+- en
+tags:
+- wake-word
+- keyword-spotting
+- openvoiceos
+- wakeforge
+- int8
+- onnx
+base_model: TigreGotico/wakehubert-tiny
+datasets:
+- TigreGotico/synthetic-wakeword-jarvis
+- TigreGotico/synthetic-wakeword-alexa
+- TigreGotico/synthetic-wakeword-ok_nabu
+- TigreGotico/synthetic-wakeword-hey_jarvis
+- TigreGotico/synthetic-wakeword-hey_marvin
+- TigreGotico/synthetic-wakeword-home_assistant
+- TigreGotico/synthetic-wakeword-hello_nabu
+- TigreGotico/synthetic-wakeword-computer
+- TigreGotico/synthetic-wakeword-wake_up
+- TigreGotico/not-wake-words-speech-en
+pipeline_tag: audio-classification
+---
+
+# WakeHuBERT wake words
+
+Ready English wake-word models for [OpenVoiceOS](https://openvoiceos.org), trained with
+[wakeforge](https://github.com/TigreGotico/wakeforge). Each model is a small GRU classifier that reads
+features from [WakeHuBERT tiny](https://huggingface.co/TigreGotico/wakehubert-tiny), a 0.64M-parameter speech
+feature extractor distilled from HuBERT-base. A model scores the last 1.5 s of audio (75 feature frames) and
+outputs a logit; its sigmoid is the probability that the window holds the wake word.
+
+The repository holds one ONNX file per word under `models/`, and `models.json`, which lists each model's word,
+featurizer, calibration, default threshold, SHA-256 and measured results. Every model carries the same facts in
+its own ONNX metadata (`wake_word`, `pretrained_featurizer`, `default_threshold`, `window_frames`, `license`,
+`training_data`, and on calibrated models `calibrated`, `calib_a` and `calib_b`).
+
+| model | word | featurizer | calibrated | default threshold |
+|---|---|---|---|---|
+| `wakehubert_jarvis` | jarvis | wakehubert-int8 | yes | 0.57 |
+| `wakehubert_alexa` | alexa | wakehubert-int8 | yes | 0.40 |
+| `wakehubert_hey_jarvis` | hey jarvis | wakehubert-int8 | yes | 0.16 |
+| `wakehubert_hey_marvin` | hey marvin | wakehubert-int8 | yes | 0.34 |
+| `wakehubert_home_assistant` | home assistant | wakehubert-int8 | yes | 0.19 |
+| `wakehubert_okay_nabu` | okay nabu | wakehubert-int8 | yes | 0.45 |
+| `wakehubert_hello_nabu` | hello nabu | wakehubert-int8 | yes | 0.49 |
+| `wakehubert_computer` | computer | wakehubert (float32) | no | 0.99 |
+| `wakehubert_hey_mycroft` | hey mycroft | wakehubert (float32) | no | 0.965 |
+| `wakehubert_wake_up` | wake up | wakehubert (float32) | no | 0.990 |
+
+## Use with OpenVoiceOS
+
+The models run in [ovos-ww-plugin-wakeforge](https://github.com/OpenVoiceOS/ovos-ww-plugin-wakeforge), which
+bundles them together with both WakeHuBERT featurizers, so nothing else is downloaded. Install the plugin and name
+the model in `mycroft.conf`:
+
+```bash
+pip install --pre ovos-ww-plugin-wakeforge
+```
+
+```json
+{
+  "listener": {
+    "wake_word": "hey_jarvis"
+  },
+  "hotwords": {
+    "hey_jarvis": {
+      "module": "ovos-ww-plugin-wakeforge",
+      "model": "wakehubert_hey_jarvis",
+      "listen": true
+    }
+  }
+}
+```
+
+A model file from this repository also loads by path: set `model` to the local `.onnx` file. The plugin reads
+the featurizer and the default threshold from the model's metadata.
+
+## Choosing a threshold
+
+Set `"threshold"` in the hotword config to trade missed wake words against false activations. A lower value
+fires more readily and falsely more often; a higher value misses more wake words and fires falsely less often.
+
+On a calibrated model the number means the same thing for every word. Its score is mapped so that a threshold of
+0.5 gives about one false activation per hour on held-out speech and noise. The default threshold is the point
+that maximises F2, which weighs recall above precision. Raising the threshold toward 0.8 or 0.9 trades recall for
+fewer false activations, and lowering it does the opposite.
+
+The uncalibrated models (`wakehubert_computer`, `wakehubert_hey_mycroft` and `wakehubert_wake_up`) output a
+probability too, but their score is not mapped to a false-activation rate. Their threshold is not a calibrated
+knob: the same number gives a different trade-off on each of them, and useful values sit close to 1.
+
+## Results
+
+Measured through the plugin at the default threshold and at 0.8. Recall is the share of test clips detected;
+false activations are counted per hour of negative audio.
+
+| model | recall at default | false activations/h at default | recall at 0.8 | false activations/h at 0.8 | recall test set |
+|---|---|---|---|---|---|
+| `wakehubert_jarvis` | 98.4% | 0.69 | 93.8% | 0.15 | 384 Picovoice recordings of real speakers |
+| `wakehubert_alexa` | 86.7% | 0.69 | 73.3% | 0.13 | 315 Picovoice recordings of real speakers |
+| `wakehubert_hey_jarvis` | 94.5% | 0.09 | 87.0% | 0.00 | synthetic test set |
+| `wakehubert_hey_marvin` | 97.4% | 0.95 | 90.7% | 0.24 | synthetic test set |
+| `wakehubert_home_assistant` | 91.1% | 0.30 | 84.2% | 0.15 | synthetic test set |
+| `wakehubert_okay_nabu` | 92.0% | 0.26 | 75.4% | 0.02 | synthetic test set |
+| `wakehubert_hello_nabu` | 80.9% | 0.39 | 65.2% | 0.02 | synthetic test set |
+
+For `wakehubert_jarvis` and `wakehubert_alexa` the false activations are counted over 46.5 h of negative audio.
+No figures are published for the uncalibrated models.
+
+## Training data
+
+The calibrated models were trained on synthetic speech only, with six edge-tts voices held out of training for
+testing. The positives are an edge-tts voice grid, further edge-tts and Google Translate TTS voices and OmniVoice
+clips, and for `wakehubert_jarvis` also voice-converted copies of edge-tts clips. Each model's `training_data`
+metadata names its own sources. Most of these clips are published in the `TigreGotico/synthetic-wakeword-<word>`
+datasets listed above. Speech from LibriSpeech train-clean-100 is mixed into training clips as background babble.
+The negatives are the wakeforge negative list and half of an AudioSet noise sample; the other half is held out.
+
+`wakehubert_computer` and `wakehubert_wake_up` were trained on synthetic speech only, from
+`TigreGotico/synthetic-wakeword-computer` and `TigreGotico/synthetic-wakeword-wake_up`, with negatives from
+`TigreGotico/not-wake-words-speech-en` and AudioSet-derived clips. `wakehubert_hey_mycroft` was trained on human
+recordings.
+
+## Calibration
+
+A calibrated model has an affine map folded into its graph, applied to the GRU's logit before the sigmoid. The map
+is fitted on a stream of held-out noise and speech that the model did not train on, so that a probability of 0.5
+falls at about one false activation per hour of that stream. The default threshold is then the F2-optimal point on
+the calibrated scale. The fitted slope and offset are in each model's `calib_a` and `calib_b` metadata.
+
+## Limitations
+
+- Five of the seven calibrated models are scored on synthetic test voices. Only `wakehubert_jarvis` and
+  `wakehubert_alexa` are scored on real speakers, so recall on real speech can be lower for the other words.
+- The calibration is fitted on about 10 h of audio with few false activations in it (2 to 14 per model), so the
+  map is extrapolated, and "0.5 is about one false activation per hour" is approximate.
+- On real jarvis recordings, `wakehubert_jarvis` peaks just above its 0.57 default, so a quiet or distant speaker
+  has little margin.
+- Similar-sounding words trigger each other's model. `wakehubert_jarvis` fires on "hey jarvis", and
+  `wakehubert_hello_nabu`, `wakehubert_hey_marvin` and `wakehubert_okay_nabu` can fire on each other's words at
+  their default thresholds. Raise the threshold when two of these models run side by side.
+- The models are English only.
+
+## License
+
+Apache-2.0. The featurizer, [TigreGotico/wakehubert-tiny](https://huggingface.co/TigreGotico/wakehubert-tiny), is
+Apache-2.0. The `synthetic-wakeword-*` and `not-wake-words-speech-en` datasets are CC BY 4.0; LibriSpeech is
+CC BY 4.0; AudioSet labels are CC BY 4.0 and its audio comes from YouTube videos under their uploaders' terms.

models.json

@@ -0,0 +1,239 @@
+{
+  "models": [
+    {
+      "name": "wakehubert_jarvis",
+      "file": "models/wakehubert_jarvis.onnx",
+      "word": "jarvis",
+      "language": "en",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.57,
+      "sha256": "b3db39a777aeb66f69523a7964dd4cae5545770494d24c0d12b1a46669e3de8c",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: an edge-tts voice grid, voice-converted copies of edge-tts clips, edge-tts and Google Translate TTS voices, OmniVoice clips, with six held-out edge-tts test voices excluded; LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.984,
+          "false_activations_per_hour": 0.69,
+          "recall_test_set": "Picovoice wake-word benchmark recordings of real speakers",
+          "recall_test_clips": 384,
+          "negative_hours": 46.5
+        },
+        "0.8": {
+          "recall": 0.938,
+          "false_activations_per_hour": 0.15,
+          "recall_test_set": "Picovoice wake-word benchmark recordings of real speakers",
+          "recall_test_clips": 384,
+          "negative_hours": 46.5
+        }
+      }
+    },
+    {
+      "name": "wakehubert_alexa",
+      "file": "models/wakehubert_alexa.onnx",
+      "word": "alexa",
+      "language": "en",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.4,
+      "sha256": "7f4b65ba036487958d180c87210e114547e8f73106317eb92a27ebc39133b4ef",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: edge-tts and Google Translate TTS voices, OmniVoice clips, with six held-out edge-tts test voices excluded; LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.867,
+          "false_activations_per_hour": 0.69,
+          "recall_test_set": "Picovoice wake-word benchmark recordings of real speakers",
+          "recall_test_clips": 315,
+          "negative_hours": 46.5
+        },
+        "0.8": {
+          "recall": 0.733,
+          "false_activations_per_hour": 0.13,
+          "recall_test_set": "Picovoice wake-word benchmark recordings of real speakers",
+          "recall_test_clips": 315,
+          "negative_hours": 46.5
+        }
+      }
+    },
+    {
+      "name": "wakehubert_hey_jarvis",
+      "file": "models/wakehubert_hey_jarvis.onnx",
+      "word": "hey jarvis",
+      "language": "en",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.16,
+      "sha256": "02851e30e8c5f81ce8dfcc8ddc8c506ad4e21e2eebb993bd3553ae0017445202",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: an edge-tts voice grid, edge-tts and Google Translate TTS voices, OmniVoice clips, with six held-out edge-tts test voices excluded; LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.945,
+          "false_activations_per_hour": 0.09,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        },
+        "0.8": {
+          "recall": 0.87,
+          "false_activations_per_hour": 0.0,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        }
+      }
+    },
+    {
+      "name": "wakehubert_hey_marvin",
+      "file": "models/wakehubert_hey_marvin.onnx",
+      "word": "hey marvin",
+      "language": "en",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.34,
+      "sha256": "1d0929a37d107032c0b0c410bba01edc01da406acb1760cc0574e0c1e9112294",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: an edge-tts voice grid, OmniVoice clips, with six held-out edge-tts test voices excluded; LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.974,
+          "false_activations_per_hour": 0.95,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        },
+        "0.8": {
+          "recall": 0.907,
+          "false_activations_per_hour": 0.24,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        }
+      }
+    },
+    {
+      "name": "wakehubert_home_assistant",
+      "file": "models/wakehubert_home_assistant.onnx",
+      "word": "home assistant",
+      "language": "en",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.19,
+      "sha256": "50a57ca721607b80d7fc198d7875248ef6b93db609026370348ab84192f95d4f",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: an edge-tts voice grid, edge-tts and Google Translate TTS voices, OmniVoice clips, with six held-out edge-tts test voices excluded; LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.911,
+          "false_activations_per_hour": 0.3,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        },
+        "0.8": {
+          "recall": 0.842,
+          "false_activations_per_hour": 0.15,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        }
+      }
+    },
+    {
+      "name": "wakehubert_okay_nabu",
+      "file": "models/wakehubert_okay_nabu.onnx",
+      "word": "okay nabu",
+      "language": "en",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.45,
+      "sha256": "41fbf1641ccd83a60c128aea27a79e7cdd8fa3612eb1bb8097b076120a8144bd",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: an edge-tts voice grid, edge-tts and Google Translate TTS voices, OmniVoice clips, with six held-out edge-tts test voices excluded; LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.92,
+          "false_activations_per_hour": 0.26,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        },
+        "0.8": {
+          "recall": 0.754,
+          "false_activations_per_hour": 0.02,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        }
+      }
+    },
+    {
+      "name": "wakehubert_hello_nabu",
+      "file": "models/wakehubert_hello_nabu.onnx",
+      "word": "hello nabu",
+      "language": "en",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.49,
+      "sha256": "ca247672badf4684de5d11b0282fbf129ab5338b17bed0c2f18fc69004de0ece",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: an edge-tts voice grid, further edge-tts voices, OmniVoice clips, with six held-out edge-tts test voices excluded; LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.809,
+          "false_activations_per_hour": 0.39,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        },
+        "0.8": {
+          "recall": 0.652,
+          "false_activations_per_hour": 0.02,
+          "recall_test_set": "synthetic test set",
+          "recall_test_clips": null,
+          "negative_hours": null
+        }
+      }
+    },
+    {
+      "name": "wakehubert_computer",
+      "file": "models/wakehubert_computer.onnx",
+      "word": "computer",
+      "language": "en",
+      "featurizer": "wakehubert",
+      "calibrated": false,
+      "default_threshold": 0.99,
+      "sha256": "4ecda088bcce9951730de9fc8b1c4f801ea208add0cbf8a1b9397ed7d8329496",
+      "license": "Apache-2.0",
+      "training_data": "TigreGotico/synthetic-wakeword-computer (CC BY 4.0); negatives: TigreGotico/not-wake-words-speech-en (CC BY 4.0) and AudioSet-derived clips",
+      "measured": null
+    },
+    {
+      "name": "wakehubert_hey_mycroft",
+      "file": "models/wakehubert_hey_mycroft.onnx",
+      "word": "hey mycroft",
+      "language": "en",
+      "featurizer": "wakehubert",
+      "calibrated": false,
+      "default_threshold": 0.965,
+      "sha256": "42b9f262c8414c5613280ab1bb8b848c47531b59b955f29bfd6902e500b53415",
+      "license": "Apache-2.0",
+      "training_data": "human recordings",
+      "measured": null
+    },
+    {
+      "name": "wakehubert_wake_up",
+      "file": "models/wakehubert_wake_up.onnx",
+      "word": "wake up",
+      "language": "en",
+      "featurizer": "wakehubert",
+      "calibrated": false,
+      "default_threshold": 0.99,
+      "sha256": "3b62701229305815fd07955f97aefb522f23689292c59fa0fbd61dd2a9cc34ee",
+      "license": "Apache-2.0",
+      "training_data": "TigreGotico/synthetic-wakeword-wake_up (CC BY 4.0); negatives: TigreGotico/not-wake-words-speech-en (CC BY 4.0) and AudioSet-derived clips",
+      "measured": null
+    }
+  ]
+}

models/wakehubert_alexa.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f4b65ba036487958d180c87210e114547e8f73106317eb92a27ebc39133b4ef
+size 467299

models/wakehubert_computer.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ecda088bcce9951730de9fc8b1c4f801ea208add0cbf8a1b9397ed7d8329496
+size 5354674

models/wakehubert_hello_nabu.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca247672badf4684de5d11b0282fbf129ab5338b17bed0c2f18fc69004de0ece
+size 467314

models/wakehubert_hey_jarvis.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02851e30e8c5f81ce8dfcc8ddc8c506ad4e21e2eebb993bd3553ae0017445202
+size 467331

models/wakehubert_hey_marvin.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d0929a37d107032c0b0c410bba01edc01da406acb1760cc0574e0c1e9112294
+size 467290

models/wakehubert_hey_mycroft.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42b9f262c8414c5613280ab1bb8b848c47531b59b955f29bfd6902e500b53415
+size 10691744

models/wakehubert_home_assistant.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50a57ca721607b80d7fc198d7875248ef6b93db609026370348ab84192f95d4f
+size 467334

models/wakehubert_jarvis.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3db39a777aeb66f69523a7964dd4cae5545770494d24c0d12b1a46669e3de8c
+size 467369

models/wakehubert_okay_nabu.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41fbf1641ccd83a60c128aea27a79e7cdd8fa3612eb1bb8097b076120a8144bd
+size 467331

models/wakehubert_wake_up.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b62701229305815fd07955f97aefb522f23689292c59fa0fbd61dd2a9cc34ee
+size 5354674