nandukmelath/nirnay-multilingual

🌍 Nirnay multilingual — typed decisions in 100 languages, on CPU

Ask choice, yes / no and rating questions about text in any of ~100 languages and get calibrated probabilities in one pass — no GPU, no text generation, order-proof, deterministic. Same API and question format as Nirnay and Laya / Jev (/v1/systemone). Write the question in English; the text can be in any language.

held-out benchmark (zero-shot)	Nirnay multilingual	Laya multilingual	Nirnay v3.1 (English)
MASSIVE intents: 51 languages, all 60 intents as options	0.386	0.217	0.076
per language vs Laya multilingual	49 wins, 1 tie, 1 loss
SIB-200 topics: 205 languages and scripts, 7 topics	0.670	0.632	0.411
per language vs Laya multilingual	105 wins, 32 ties, 68 losses
option-order flip rate on these suites	0.000	0.185	0.000
CPU time per case on these suites	89 ms	138 ms

Laya multilingual (laya 0.4.14, its mmBERT-base checkpoint, its own API) ran on the same machine on byte-identical cases. Neither model trained on MASSIVE or SIB-200. MASSIVE asks one 60-way question per request with no examples — hard for any zero-shot model (Nirnay's absolute score is modest; Laya's fixed option budget struggles more). Examples from MASSIVE: Hindi 0.350 vs 0.175, Spanish 0.400 vs 0.175, Malayalam 0.463 vs 0.087, Tamil 0.412 vs 0.125, Urdu 0.438 vs 0.100, Bengali 0.450 vs 0.163; Chinese (simplified) is the one loss, 0.338 vs 0.412.

Which model?

• English text: use nandukmelath/nirnay (v3.1) — more accurate in English on most tasks, faster and half the size.
• Any other language, or mixed-language traffic: use this model.

Quick start

pip install git+https://github.com/nandukmelath/nirnay

import nirnay

m = nirnay.load("nandukmelath/nirnay-multilingual")

q = {"intent": {"type": "choice", "instructions": "What does the user want?",
                "criteria": ["set an alarm", "check the weather", "play music", "order food", "book a taxi"]},
     "urgent": {"type": "noul", "instructions": "Is this request urgent?"}}

m.predict({"message": "¿Me puedes despertar mañana a las seis y media?"}, q)["answers"]["intent"]["choice"]
# 'set an alarm' (confidence 1.00)
m.predict({"message": "എനിക്ക് ഇപ്പോൾ തന്നെ എയർപോർട്ടിലേക്ക് ഒരു ടാക്സി വേണം"}, q)["answers"]
# intent 'book a taxi' (0.98), urgent P(yes) 0.65

Real outputs of this model: Spanish → set an alarm, Hindi "कल दिल्ली में बारिश होगी क्या?" → check the weather, Malayalam → book a taxi, German "Spiel bitte etwas ruhige Jazzmusik" → play music, Arabic "أريد أن أطلب بيتزا كبيرة" → order food. predict_batch, learn() (few-shot memory), the HTTP server and the MCP server work exactly as in Nirnay.

English benchmarks (same 400 cases per task as the English model)

task	multilingual	Nirnay v3.1	best Laya checkpoint
AG News (4 topics)	0.943	0.925	0.953
DAIR emotion (6)	0.485	0.595	0.600
Banking77 (77 intents)	0.477	0.598	0.492
support triage (10 queues)	0.425	0.407	0.540
Enron spam	0.995	0.995	0.998
phishing	0.975	0.973	0.993
jailbreak (toxic-chat)	0.755	0.647	0.805
toxicity (toxic-chat)	0.657	0.688	0.535
RAG passage relevance	0.660	0.680	0.672
model routing (domain)	0.857	0.875	0.659
SST-5 (5-level rating)	0.480	0.490	0.470
SMS spam	0.868	0.790	0.938

Against the best of Laya's three checkpoints in English: 3 wins (toxicity, model routing, SST-5), 1 tie, 8 losses.

speed (warm p50, same CPU)	1 question	10 questions	50 questions
Nirnay multilingual	146 ms	297 ms	829 ms
Nirnay v3.1 (English)	129 ms	232 ms	655 ms
Laya at its fastest (any checkpoint, eager or ONNX)	159 ms	1,389 ms	6,730 ms

Determinism: repeat calls byte-identical; across 1/4/8/16 CPU threads no decision changed (probabilities differ by at most 0.0001). Few-shot learn(), Banking77: 0.478 with no examples → 0.768 with 8 per intent (0.745 with 16). Full tables: REPORT.md.

How it works

The same architecture as Nirnay (deep fusion: the encoder's top 4 layers run over [state ; questions ; options] with an asymmetric mask, so the text is read once per call and options never see each other), on intfloat/multilingual-e5-base (XLM-RoBERTa, 100 languages). Trained on a CPU with proper scoring rules: ~1.5 epochs over 385k typed questions on both sockets of a 2014 workstation.

Limitations

• Absolute accuracy on 60-way intents is modest (0.386 averaged over 51 languages) — it is zero-shot with no examples. For production intents, add a few labelled examples per intent with learn().
• English is better served by v3.1 (emotion 0.485 vs 0.595, Banking77 0.477 vs 0.598).
• SIB-200 is a narrower win than MASSIVE (105 wins, 32 ties, 68 losses per language); some low-resource scripts stay weak for both models.
• 1.2 GB on disk, +1.5 GB RAM — XLM-R's 250k-token vocabulary (the English model is 527 MB).
• Wording still matters for concepts it rarely trained on; the questions are best written in English.

Training data

Everything in Nirnay v3.1's mix (see its card), plus: MTOP assistant intents in 6 languages (tasksource/mtop, CC-BY-SA-4.0 — a different dataset from MASSIVE, with similar assistant intents), PAWS-X paraphrases in 6 languages, machine-translated MNLI in 26 languages (MoritzLaurer/multilingual-NLI-26lang-2mil7), TextDetox toxicity in 14 languages (OpenRAIL++), language identification in 20 languages (papluca), multilingual Amazon reviews and multilingual tweet sentiment. Held out: MASSIVE, SIB-200, XNLI, and everything v3.1 holds out (emotion, Banking77, toxic-chat, gsm8k, mbpp, SST-5, the UCI SMS collection, all jailbreak datasets); any text appearing in a benchmark case was removed.

Some training datasets carry research-only or non-commercial terms (for example the multilingual Amazon reviews, MS MARCO, RACE, Yahoo Answers, Yelp reviews). The code and weights are released under Apache-2.0; check those datasets' terms before commercial use.

Built on intfloat/multilingual-e5-base (MIT). The benchmark cases reproduce the application benchmark of Laya by Nandakishor M (Convai Innovations), whose open model inspired this project.