Horizon-Labs/multilingual-keyword-vs-question-small

Multilingual Keyword Query vs Natural Language Classifier (small, 141M)

Tells whether a search query is a keyword query ("cheap flights berlin paris") or natural language, a question or a statement ("how do I get from berlin to paris"), in about 50 languages. Use it to send keyword queries to a lexical retriever (BM25) and natural-language queries to a dense retriever or a QA pipeline. A multilingual drop-in replacement for shahrukhx01/bert-mini-finetune-question-detection (used in Haystack's query classifier): same labels, LABEL_0 = keywords, LABEL_1 = natural language. Built on mmBERT-small, Apache-2.0, ONNX for CPU and the browser (transformers.js) included. onnx/model_quantized.onnx (int8 embeddings, 268 MB) gives the same label as fp32 for 100.0% of 400 test texts. For question vs statement, see multilingual-question-vs-statement-small.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Horizon-Labs/multilingual-keyword-vs-question-small")
print(clf(["cheap flights berlin paris", "how do I get from berlin to paris", "billige flüge berlin paris", "wie komme ich nach paris"]))
# LABEL_0 = keyword query, LABEL_1 = natural language (question or statement)

In Haystack, use it wherever shahrukhx01/bert-mini-finetune-question-detection is configured; the label names are the same.

Evaluation

Labels are objective (by construction), and no test text was used for training.

• Keyword vs question queries: 9,399 held-out search queries in 47 languages generated by Qwen3.8-27B from web passages, each passage giving a keyword query and a natural-language question (question mark removed).
• Keyword queries vs web sentences: the held-out keyword queries against 3,944 held-out web sentences from FineWeb-2 / FineWeb (questions and statements, final punctuation removed, half lowercased).

model	keyword vs question queries: accuracy	mean balanced accuracy over languages	English	keyword queries vs web sentences: accuracy	mean balanced accuracy over languages	English	note
this model (141M)	0.992	0.992	0.995	0.986	0.985	0.988	66 languages
shahrukhx01/bert-mini-finetune-question-detection (11M)	0.536	0.536	0.965	0.577	0.544	0.950	English (bert-mini); Haystack's default

• The table shows the released checkpoint. Means over 2 training seeds (queries .992 / .992, keywords vs sentences .987 / .986).
• Lowest languages (keywords vs sentences, balanced accuracy): fil_Latn 0.95, nld_Latn 0.96, hun_Latn 0.96, vie_Latn 0.96, mar_Deva 0.97, rus_Cyrl 0.97.

Training

• 160,525 examples: keyword and question queries generated for our multilingual reranker, plus real web sentences from FineWeb-2 / FineWeb (ODC-BY). No test queries or sentences.
• mmBERT-small, two classes, max length 128 tokens. Code: code/ in this repository.

Limitations

• The keyword queries are model-generated; real users mix styles ("berlin paris train how long"). Short natural phrases such as product names or titles are hard to tell from keywords.