OpenVoiceOS/ovos-m2v-intents-multilingual

OVOS intent classifier, multilingual

A static-embedding intent classifier for OpenVoiceOS. It maps an utterance to one of 221 intent labels of the form <skill_id>:<intent_name>, the names the OVOS m2v pipeline registers at runtime. It is the default model of ovos-m2v-pipeline and the fallback for every language without a model of its own. English also has a smaller English-only model, ovos-m2v-intents-en.

Inference is a vector lookup, an average and one linear layer, so it runs on a CPU. The embedding table is 500,353 tokens by 256 dimensions in float32, about 530 MB on disk and in memory.

Use

The OVOS pipeline loads this repository by default. To load it directly:

from model2vec.inference import StaticModelPipeline

model = StaticModelPipeline.from_pretrained("OpenVoiceOS/ovos-m2v-intents-multilingual")
model.predict(["set a timer for ten minutes"])

labels.json lists the 221 labels the model was trained on. The pipeline reads it and matches only those labels.

Smaller versions and ONNX

The repository root holds the float32 model, which the OVOS pipeline loads by default. Three more copies sit in folders beside it. In fp16 and int8 only the embedding table changes precision; the labels and the classifier layer are the same.

Folder	Format	Size	Gold, all (12,908 rows)	Gold, not needs_manual (2,412 rows)
root	model2vec, float32	532 MB	0.8464	0.8478
fp16	model2vec, float16	276 MB	0.8464	0.8478
int8	model2vec, int8	148 MB	0.8458	0.8503
onnx	ONNX, float32	513 MB	same predictions as the root model

To use a smaller copy in OVOS, set subfolder:

{"intents": {"ovos-m2v-pipeline": {
  "model": "OpenVoiceOS/ovos-m2v-intents-multilingual",
  "subfolder": "int8"}}}

In Python, download one folder and load it:

from huggingface_hub import snapshot_download
from model2vec.inference import StaticModelPipeline

path = snapshot_download("OpenVoiceOS/ovos-m2v-intents-multilingual", allow_patterns=["int8/*"])
model = StaticModelPipeline.from_pretrained(f"{path}/int8")

onnx/model.onnx runs without model2vec. It takes input_ids and attention_mask (int64, batch by tokens) and returns probabilities (float32, batch by 221), in the order of head_config.classes in onnx/config.json. Tokenize with onnx/tokenizer.json without special tokens and keep at most 512 tokens:

import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("onnx/tokenizer.json")
classes = json.load(open("onnx/config.json"))["head_config"]["classes"]
ids = tok.encode("set a timer for ten minutes", add_special_tokens=False).ids[:512]
sess = ort.InferenceSession("onnx/model.onnx")
probs = sess.run(None, {"input_ids": np.array([ids]), "attention_mask": np.ones((1, len(ids)), np.int64)})[0]
print(classes[probs[0].argmax()])

One-language versions

For devices with little memory, such as a Raspberry Pi, a folder per language holds an int8 copy of this model with its vocabulary cut to that language. It keeps the tokens the language's training sentences use and the tokens of its 50,000 most common words (20,000 for Spanish; Galician and Basque keep only the tokens of their training sentences). The embeddings of the kept tokens, the labels and the classifier layer are unchanged; nothing is retrained. A word whose tokens were cut is not seen, so a sentence with rare words can score lower than the gold sentences show.

Gold rows are the language's rows of the gold set; the last column is the full float32 model on the same rows.

Folder	Language	Size	Gold, all	Rows	Full model
pt	Portuguese (pt-PT, pt-BR)	13 MB	0.879	1,183	0.885
es	Spanish	8.3 MB	0.852	1,032	0.846
ca	Catalan	12 MB	0.852	608	0.854
da	Danish	8.8 MB	0.854	629	0.857
de	German	12 MB	0.834	614	0.834
nl	Dutch	12 MB	0.849	636	0.852
sv	Swedish	11 MB	0.873	715	0.877
gl	Galician	2.4 MB	0.831	608	0.854
eu	Basque	2.5 MB	0.854	615	0.859

From ovos-m2v-pipeline 0.36.0a2, a models entry names the folder for its language:

{"intents": {"ovos-m2v-pipeline": {"models": {
  "pt": {"model": "OpenVoiceOS/ovos-m2v-intents-multilingual", "subfolder": "pt"}}}}}

Earlier models

The model trained on the same data in 128 shards taken in order, which scores 0.8155 on all gold rows, is the tag v7-preview2. Set "revision": "v7-preview2" to load it.

The model before it, with 210 labels in the older label names, is revision 145a1bd94acb2620d9aa92c82567cf2bbb55f0a2. In the OVOS configuration:

{"intents": {"ovos-m2v-pipeline": {
  "model": "OpenVoiceOS/ovos-m2v-intents-multilingual",
  "revision": "145a1bd94acb2620d9aa92c82567cf2bbb55f0a2"}}}

In Python:

from huggingface_hub import snapshot_download
from model2vec.inference import StaticModelPipeline

path = snapshot_download("OpenVoiceOS/ovos-m2v-intents-multilingual",
                         revision="145a1bd94acb2620d9aa92c82567cf2bbb55f0a2")
model = StaticModelPipeline.from_pretrained(path)

Training

Training data	OpenVoiceOS/ovos-intents, revision v7-preview2 (commit e03197a)
Corpus	4,952,006 training rows, 12,908 gold rows, 221 labels, 53 training and 51 gold locales
Base model	minishlab/potion-multilingual-128M at 73908c34
Classifier	model2vec 0.9.0 StaticModelForClassification, embeddings trained with the head
Optimiser	AdamW, learning rate 0.001, cross-entropy loss
Batch and epochs	batch 512, 3 epochs, the whole training set shuffled in every epoch, seed 13
Held-out split	10% of each label's templates, chosen by a hash of the template text; every row of a held-out template is left out of training, and a label keeps at least one training template

The held-out side has 14,969 templates. 3,004 of their rows repeat a training utterance and are left out of the score, which leaves 457,445 rows. Training used the other 135,695 templates, 4,491,557 rows.

Results

Measure	Rows	Accuracy
Gold sentences, all	12,908	0.8464
Gold sentences not flagged needs_manual	2,412	0.8478
Held-out templates	457,445	0.9838
Labels shared with the 210-label model, this model	5,869	0.8347
Labels shared with the 210-label model, that model	5,869	0.5152

The gold sentences are the skills' own golden utterances, not generated from the templates the model trains on. 10,496 of the 12,908 are flagged needs_manual; they are scored like the others. The last two rows compare this model and the 210-label model on the gold rows whose label both can emit, 85 labels.

Gold accuracy per language:

Locale	Gold rows	Accuracy
en-US	1,506	0.840
pt-BR	655	0.893
nl-NL	636	0.852
da-DK	629	0.857
pt-PT	623	0.888
it-IT	619	0.859
sv-SE	617	0.867
es-ES	615	0.852
eu-ES	615	0.859
de-DE	614	0.834
ca-ES	608	0.854
gl-ES	608	0.854
fr-FR	601	0.812
ru-RU	540	0.839
kab	521	0.754
pl-PL	502	0.813
es-CO	491	0.839
fa-IR	487	0.801
oc-FR	249	0.827
cs-CZ	211	0.872
hu-HU	197	0.822
fi-FI	149	0.846
sv-FI	146	0.945
tr-TR	118	0.856
el-GR	99	0.889
an	45	0.689
ro-RO	38	0.947
lt-LT	19	0.947

Languages with fewer than 10 gold rows are not meaningful and are left out of the table: bg-BG (9), he-IL (9), hr-HR (9), id-ID (9), ja-JP (9), nb-NO (9), sk-SK (9), sl-SI (9), th-TH (9), ar-XA (6), et-EE (6), fil-PH (6), ko-KR (6), lv-LV (6), ms-MY (6), uk-UA (6), vi-VN (6), zh-CN (6), ar-XX (3), cmn-CN (3), is-IS (3), nn-NO (3), zsm-MY (3).

Limits

• Labels are skewed. Templates that multiply slot values produce most of the rows. ovos-skill-date-time:what_time_will_it_be is 18.9% of the corpus and ovos-skill-pokepedia:battle 15.0%; in Kabyle the first is 78.5% of the rows and in Polish 69.4%. The median label has 479 rows and the smallest 10.
• Place names are not declined. A {location} slot is filled with the name as written, so a Polish row reads "we Irlandia" where a speaker says "w Irlandii". The model has seen few correctly inflected place names in languages that decline them.
• The gold sentences are model-written. Every gold sentence in the fleet was written by a language model. needs_manual marks rows meant for a human check, and whether that check happened is not recorded, so the gold figures measure agreement with model-written sentences, not with recorded users.

License

Apache-2.0. The base model is MIT.