Calibrated int8 wake-up models in Spanish, German, Dutch and Italian

README.md

@@ -3,6 +3,10 @@ license: apache-2.0
 library_name: onnx
 language:
 - en
+- es
+- de
+- nl
+- it
 tags:
 - wake-word
 - keyword-spotting
@@ -34,12 +38,16 @@ datasets:
 - TigreGotico/synthetic-wakeword-hey_k9
 - TigreGotico/synthetic-wakeword-hey_scout
 - TigreGotico/not-wake-words-speech-en
+- TigreGotico/synthetic-wakeword-despierta
+- TigreGotico/synthetic-wakeword-aufwachen
+- TigreGotico/synthetic-wakeword-wakker_worden
+- TigreGotico/synthetic-wakeword-sveglia
 pipeline_tag: audio-classification
 ---
 
 # WakeHuBERT wake words
 
-Ready English wake-word models for [OpenVoiceOS](https://openvoiceos.org), trained with
+Ready wake-word models for [OpenVoiceOS](https://openvoiceos.org), trained with
 [wakeforge](https://github.com/TigreGotico/wakeforge). Each model is a small GRU classifier that reads
 features from [WakeHuBERT tiny](https://huggingface.co/TigreGotico/wakehubert-tiny), a 0.64M-parameter speech
 feature extractor distilled from HuBERT-base. A model scores the last 1.5 s of audio (75 feature frames) and
@@ -78,6 +86,10 @@ its own ONNX metadata (`wake_word`, `pretrained_featurizer`, `default_threshold`
 | `wakehubert_hey_stemcom` | hey stemcom | wakehubert-int8 | yes | 0.28 |
 | `wakehubert_computer` | computer | wakehubert (float32) | no | 0.99 |
 | `wakehubert_hey_mycroft` | hey mycroft | wakehubert-int8 | yes | 0.18 |
+| `wakehubert_despierta` | despierta (Spanish) | wakehubert-int8 | yes | 0.05 |
+| `wakehubert_aufwachen` | aufwachen (German) | wakehubert-int8 | yes | 0.43 |
+| `wakehubert_wakker_worden` | wakker worden (Dutch) | wakehubert-int8 | yes | 0.16 |
+| `wakehubert_sveglia` | sveglia (Italian) | wakehubert-int8 | yes | 0.08 |
 
 ## Use with OpenVoiceOS
 
@@ -207,10 +219,14 @@ false activations are counted per hour of negative audio.
 | `wakehubert_hey_potato` | 88.9% | 0.34 | 80.6% | 0.02 | 360 clips in held-out edge-tts voices only, without OmniVoice test clips |
 | `wakehubert_hey_stemcom` | 89.0% | 0.21 | 83.1% | 0.06 | 337 clips of OmniVoice and held-out edge-tts voices |
 | `wakehubert_hey_mycroft` | 97.4% | 1.57 | 84.8% | 0.24 | 285 clips in held-out edge-tts voices and 379 Kokoro clips converted to unseen speakers |
+| `wakehubert_despierta` | 95.4% | 0.39 | 92.4% | 0.16 | 370 Spanish OmniVoice clips with unseen seeds |
+| `wakehubert_aufwachen` | 99.2% | 0.16 | 94.6% | 0.03 | 390 German OmniVoice clips with unseen seeds |
+| `wakehubert_wakker_worden` | 98.2% | 0.58 | 94.4% | 0.06 | 390 Dutch OmniVoice clips with unseen seeds |
+| `wakehubert_sveglia` | 98.7% | 0.19 | 96.0% | 0.03 | 222 Italian OmniVoice clips with unseen seeds |
 
 The false activations are counted over 46.5 h of negative audio (speech, non-speech and household audio). The
 held-out synthetic voices are text-to-speech voices that no training clip uses, converted to the voices of
-speakers who appear in no training clip. No figures are published for the uncalibrated model.
+speakers who appear in no training clip. For the localised wake-up models (the words with a language in brackets in the table above), the false activations are counted over the 31.1 h of speech and non-speech audio without the household audio, and recall is measured on clips of OmniVoice voices drawn from seeds that no training clip uses. No figures are published for the uncalibrated model.
 
 ## Training data
 
@@ -222,6 +238,8 @@ multi-engine, edge-tts and Piper (LibriTTS-R speakers) clips of its dataset and
 datasets listed above. Speech from LibriSpeech train-clean-100 is mixed into training clips as background babble.
 The negatives are the wakeforge negative list and half of an AudioSet noise sample; the other half is held out.
 
+The localised wake-up models were trained on synthetic speech only: OmniVoice clips with no reference speaker, one seed per clip, kept when a speech recogniser transcript matched the phrase (or, for languages no recogniser covers, when duration, level and speech-activity checks passed), together with the clips of the phrase's `TigreGotico/synthetic-wakeword-<word>` dataset, which also holds the kept OmniVoice clips and their held-out test split. Speech from LibriSpeech train-clean-100 is mixed in as background babble, and the negatives are the same as above.
+
 `wakehubert_computer` was trained on synthetic speech only, from `TigreGotico/synthetic-wakeword-computer`, with
 negatives from `TigreGotico/not-wake-words-speech-en` and AudioSet-derived clips.
 
@@ -234,7 +252,7 @@ the calibrated scale. The fitted slope and offset are in each model's `calib_a`
 
 ## Limitations
 
-- Sixteen of the twenty-three calibrated models are scored on held-out synthetic voices, so recall on real speech can
+- Twenty of the twenty-seven calibrated models are scored on held-out synthetic voices, so recall on real speech can
   be lower for those words. `wakehubert_jarvis` and `wakehubert_alexa` are scored on the Picovoice recordings,
   `wakehubert_hey_chatterbox` and `wakehubert_hey_floyd` on OVOS community recordings, and `wakehubert_marvin`,
   `wakehubert_sheila` and `wakehubert_stop` on Speech Commands, all real speakers.
@@ -253,7 +271,9 @@ the calibrated scale. The fitted slope and offset are in each model's `calib_a`
   sometimes on "hey floyd", `wakehubert_computer` and `wakehubert_hey_computer` on each other's words, and
   `wakehubert_sheila` on "computer", all at their default
   thresholds. Raise the threshold when two of these models run side by side.
-- The models are English only.
+- The models are English except the localised wake-up models, whose language is in brackets in the table above.
+  Their negative audio is English speech and non-speech audio; on read speech in their own language (the FLEURS
+  development set) at the default threshold they gave: `wakehubert_despierta` 0.00 per hour over 1.4 h; `wakehubert_aufwachen` 0.79 per hour over 1.3 h; `wakehubert_wakker_worden` 0.00 per hour over 0.5 h; `wakehubert_sveglia` 1.30 per hour over 1.5 h.
 
 ## License
 

models.json

@@ -656,6 +656,118 @@
           "negative_hours": 46.5
         }
       }
+    },
+    {
+      "name": "wakehubert_despierta",
+      "file": "models/wakehubert_despierta.onnx",
+      "word": "despierta",
+      "language": "es",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.05,
+      "sha256": "ff50e507a3aed854970655b0244805cbcbf9b103e07aedd97dbd636d4a3b5c38",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: OmniVoice clips with no reference speaker, one seed per clip, each checked against a speech recogniser transcript, and the OmniVoice clips of TigreGotico/synthetic-wakeword-despierta (CC BY 4.0); LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.9541,
+          "false_activations_per_hour": 0.39,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 370,
+          "negative_hours": 31.1
+        },
+        "0.8": {
+          "recall": 0.9243,
+          "false_activations_per_hour": 0.16,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 370,
+          "negative_hours": 31.1
+        }
+      }
+    },
+    {
+      "name": "wakehubert_aufwachen",
+      "file": "models/wakehubert_aufwachen.onnx",
+      "word": "aufwachen",
+      "language": "de",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.43,
+      "sha256": "bd1e0030d1405fd0e7cb9c25e5d1917e7df0054543fef49d7cde5f42883f5a52",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: OmniVoice clips with no reference speaker, one seed per clip, each checked against a speech recogniser transcript, and the OmniVoice clips of TigreGotico/synthetic-wakeword-aufwachen (CC BY 4.0); LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.9923,
+          "false_activations_per_hour": 0.16,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 390,
+          "negative_hours": 31.1
+        },
+        "0.8": {
+          "recall": 0.9462,
+          "false_activations_per_hour": 0.03,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 390,
+          "negative_hours": 31.1
+        }
+      }
+    },
+    {
+      "name": "wakehubert_wakker_worden",
+      "file": "models/wakehubert_wakker_worden.onnx",
+      "word": "wakker worden",
+      "language": "nl",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.16,
+      "sha256": "b1c090730190f091297fac7e07725c501b2045fd71b15929fa37aa38f5fecad7",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: OmniVoice clips with no reference speaker, one seed per clip, each checked against a speech recogniser transcript, and the OmniVoice clips of TigreGotico/synthetic-wakeword-wakker_worden (CC BY 4.0); LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.9821,
+          "false_activations_per_hour": 0.58,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 390,
+          "negative_hours": 31.1
+        },
+        "0.8": {
+          "recall": 0.9436,
+          "false_activations_per_hour": 0.06,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 390,
+          "negative_hours": 31.1
+        }
+      }
+    },
+    {
+      "name": "wakehubert_sveglia",
+      "file": "models/wakehubert_sveglia.onnx",
+      "word": "sveglia",
+      "language": "it",
+      "featurizer": "wakehubert-int8",
+      "calibrated": true,
+      "default_threshold": 0.08,
+      "sha256": "d799d6e973e145d781b435ab4ee0f0dcb60afce549c3bc40e42989f33f62b74f",
+      "license": "Apache-2.0",
+      "training_data": "synthetic only: OmniVoice clips with no reference speaker, one seed per clip, each checked against a speech recogniser transcript, and the OmniVoice clips of TigreGotico/synthetic-wakeword-sveglia (CC BY 4.0); LibriSpeech train-clean-100 (CC BY 4.0) mixed in as background babble; negatives from wakeforge datasets/train.csv and the even half of an AudioSet noise sample",
+      "measured": {
+        "default": {
+          "recall": 0.9865,
+          "false_activations_per_hour": 0.19,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 222,
+          "negative_hours": 31.1
+        },
+        "0.8": {
+          "recall": 0.9595,
+          "false_activations_per_hour": 0.03,
+          "recall_test_set": "held-out synthetic voices: OmniVoice clips whose seeds no training clip uses",
+          "recall_test_clips": 222,
+          "negative_hours": 31.1
+        }
+      }
     }
   ]
 }

models/wakehubert_aufwachen.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd1e0030d1405fd0e7cb9c25e5d1917e7df0054543fef49d7cde5f42883f5a52
+size 467388

models/wakehubert_despierta.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff50e507a3aed854970655b0244805cbcbf9b103e07aedd97dbd636d4a3b5c38
+size 467394

models/wakehubert_sveglia.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d799d6e973e145d781b435ab4ee0f0dcb60afce549c3bc40e42989f33f62b74f
+size 467391

models/wakehubert_wakker_worden.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1c090730190f091297fac7e07725c501b2045fd71b15929fa37aa38f5fecad7
+size 467402