| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-classification |
| language: [multilingual, en, de, fr, es, pt, it, nl, sv, da, nb, ru, pl, tr, ar, he, fa, ur, hi, bn, ta, te, kn, ml, th, vi, id, ms, tl, ja, ko, zh, el, hu, fi, ro, sq, sl, sw, af, cy, am, hy, ka, km, my, mn, lv, is, az, jv] |
| tags: [laya, multilingual, mmbert, system-one, calibrated-decisions, rlcd, classification, routing, guardrails, moderation, commercial-use] |
| --- |
| |
| <p align="center"> |
| <img src="https://huggingface.co/convaiinnovations/laya/resolve/main/assets/logo-mark.png" alt="" width="72" /> |
| </p> |
|
|
| # Laya Multilingual |
|
|
| Non-autoregressive **System 1 decision model** covering 100+ languages. Give it a **state** |
| (text, email, ticket, or JSON) and **typed questions**; it returns typed answers with |
| probabilities in a single forward pass. No text generation, so nothing to parse and nothing to |
| hallucinate. |
|
|
| Part of the [Laya family](https://huggingface.co/convaiinnovations/laya) — **use this checkpoint |
| for anything that is not English.** |
|
|
| | checkpoint | encoder | params | context | use it for | |
| |---|---|---|---|---| |
| | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) | ModernBERT-large | 421M | 512 | English | |
| | **`convaiinnovations/laya-multilingual`** (this repo) | mmBERT-base | 322M | 1024 (up to 8,192) | 100+ languages, ~2x faster | |
| | [`convaiinnovations/laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions) | ModernBERT-large | 421M | 1024 | the typed-decisions workflows | |
|
|
| > **Long documents: `laya-multilingual` reads up to 8,192 tokens.** It ships with a 1,024-token limit that cuts long documents off, so pass `max_len=8192` for them: |
| > |
| > ```python |
| > import laya |
| > agent = laya.load("convaiinnovations/laya-multilingual") |
| > result = agent.predict(long_document, questions, max_len=8192) |
| > ``` |
| > |
| > In the table below, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary (8 to 17 of 20), so check long-document accuracy on your own data. Short inputs give identical answers with `max_len=8192`, and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU. |
|
|
| <p align="center"> |
| <img src="https://raw.githubusercontent.com/NandhaKishorM/laya/main/assets/long_context_8192.png" alt="laya-multilingual long-document accuracy by document length" width="100%" /> |
| </p> |
|
|
| ## Quickstart |
|
|
| ```bash |
| pip install laya |
| ``` |
|
|
| ```python |
| import laya |
| |
| agent = laya.load("convaiinnovations/laya-multilingual") |
| result = agent.predict( |
| {"body": "मुझसे इनवॉइस 4411 के लिए दो बार शुल्क लिया गया। कृपया आज ही धनवापसी करें।"}, |
| {"department": {"type": "choice", "instructions": "Which team should handle `body`?", |
| "criteria": {"billing": "invoices, payments, refunds", |
| "technical": "bugs and outages", "sales": "pricing"}}, |
| "refund_requested": {"type": "noul", "instructions": "Does the sender ask for money back?"}}, |
| ) |
| print(result["answers"]["department"]["choice"]) # billing |
| ``` |
|
|
| ### Let the Router choose |
|
|
| ```python |
| from laya import Router |
| |
| router = Router() |
| router.predict({"body": "I was charged twice"}, questions) # -> laya |
| router.predict({"body": "二重に請求されました"}, questions) # -> laya-multilingual |
| ``` |
|
|
| The default `Router()` keeps both `english` and this checkpoint resident, so a |
| mixed workload no longer swaps checkpoints on every language change. For a server, load them up |
| front so even the first request of each language is just a forward pass: |
|
|
| ```python |
| router = Router() |
| router.preload(["english", "multilingual"]) # both resident; no swap at request time |
| ``` |
|
|
| `router.attach("multilingual", agent)` registers an `Agent` you already built, so a process that |
| loaded this checkpoint directly can hand it to the router instead of loading it twice. |
|
|
| More text reaches this checkpoint than script alone would send: plain-ASCII Spanish, Italian, Portuguese and French |
| (accents stripped by mail clients and ticket systems), Brazilian Portuguese support text, and any |
| script the router has no range for. If you already run a language-identification model, pass its |
| answer with `router.predict(state, questions, lang_guess=code_or_callable)`. |
|
|
| It also receives CJK requests that contain Latin brand names, romanized |
| Bangla, and Azerbaijani. On 20,000 English texts, at most 5 English sentences move, all quoting long |
| native-script names. |
|
|
| Routing is decided from the script of the input, **before** the forward pass — because the |
| model's confidence gives no warning when a checkpoint cannot read its input (see below). |
|
|
| > **If `laya.load()` hangs:** `transformers` probes for TensorFlow at import, and when TF is |
| > installed its abseil runtime can deadlock model construction. Run with `USE_TF=0`. |
| |
| ## Why this checkpoint exists |
| |
| Measured across **all 51 MASSIVE languages**, intent classification with 20 options |
| (random = 0.050), both checkpoints answering byte-identical questions: |
| |
| | | `laya` (English) | **`laya-multilingual`** | |
| |---|---|---| |
| | macro accuracy | 0.227 | **0.366** | |
| | macro ECE | 0.733 | **0.387** | |
| | languages clearing 3x random | 23 / 51 | **45 / 51** | |
| |
| The English checkpoint does not degrade gracefully outside English — it collapses, and stays |
| confident while doing so. Khmer: **0.000 accuracy at 0.952 confidence**. Hebrew 0.060, Armenian |
| 0.050 (exactly random), Bengali 0.080 — all reported with 0.89–0.96 confidence. Its mean |
| confidence never drops below 0.885 at any accuracy level, so confidence gating cannot catch it. |
| |
| Per-language, this checkpoint turns near-random into usable: Arabic 0.110 → **0.400**, Bengali |
| 0.080 → **0.290**, Azerbaijani 0.100 → **0.300**, Hindi 0.100 → **0.387**, Korean 0.110 → |
| **0.490**, Turkish 0.140 → **0.437**. |
| |
| ### XNLI (15 languages) |
| |
| | | `laya` | **`laya-multilingual`** | |
| |---|---|---| |
| | English | **0.860** | 0.843 | |
| | 14 other languages | 0.521 | **0.731** | |
| |
| ### Speed — it is also the faster checkpoint |
| |
| | questions per call | `laya` | **`laya-multilingual`** | |
| |---|---|---| |
| | 1 | 39.5 ms | **32.8 ms** | |
| | 10 | 158.6 ms (15.9 ms/q) | **72.3 ms (7.2 ms/q)** | |
| | 50 | 771 ms | **337 ms (6.8 ms/q)** | |
| |
| 103–332 questions/sec batched on one T4, despite a 256k vocabulary — the 768-dim / 22-layer |
| encoder is cheaper per token than 1024-dim / 28-layer, and the gap widens with batch size. |
| |
| ## Architecture |
| |
| - **Backbone** mmBERT-base (307M, bidirectional, 22 layers, hidden 768, 256k vocab) + a decision |
| head trained from scratch: 2 transformer layers, an option-marker scorer, and an act/escalate |
| head. 322M total. |
| - **Option markers** every option is scored at its own `[MASK]` token, then softmaxed over that |
| question's options — so the answer space is defined per request, with no retraining. |
| - **Budget** 1024 tokens per question, of which 256 go to the question and its options. |
| - Trained from scratch with RLCD: 15,987 updates, 4 epochs, ~4.97 h. |
| |
| ## Limits |
| |
| - **Ships uncalibrated.** `temperature = [1.0, 1.0, 1.0]` with no per-option-count buckets. It is |
| systematically over-confident (mean confidence 0.75–0.83 against much lower accuracy). Refitting |
| one temperature per (question type, option count) on held-out data moves mean ECE |
| **0.314 → 0.106**. Do this on your own data before trusting the probabilities. |
| - **Weaker on English** than the English checkpoint: 0.619 vs 0.684 macro across English suites. |
| Route rather than replace. |
| - **Near chance on typed-decisions zero-shot** — 0.342, against a 0.318 random and 0.461 |
| majority-class baseline. Fine-tune for a specific workflow; that is where the capability comes |
| from. |
| - **Keep `choice` questions under ~20 options.** Options share the fixed 256-token head budget, so |
| a very large label space leaves only a few tokens per label and accuracy falls off sharply. |
| - Low-resource languages are weak, not fixed: Swahili 0.210, Tamil 0.250, Amharic 0.110. |
| - Ordinal `score` questions are the weakest primitive (SST-5 0.282), and this checkpoint has a |
| **measured position bias on them**: it rarely picks the first-listed level, in any language, |
| including English (0 of 290 in one independent run, |
| [#131](https://github.com/NandhaKishorM/laya/issues/131)). For English score questions use the |
| English checkpoint. For other languages, validate score outputs on your own data first. |
| - **`noul` can under-report "true" here.** On a clearly positive input, one measurement put |
| `P(true)` at about 0.5 while the negative case was correctly near 0 |
| ([#156](https://github.com/NandhaKishorM/laya/issues/156)). If `noul` answers look weak, the same |
| question as a two-option `choice` with neutral keys (`A` / `B`) and yes/no descriptions is a |
| useful check. |
| |
| ## Links |
| |
| - **Docs** https://nandhakishorm.github.io/laya/ |
| - **Hub / family** https://huggingface.co/convaiinnovations/laya |
| - **GitHub** https://github.com/NandhaKishorM/laya · full benchmark data on the `research` branch |
| - **PyPI** https://pypi.org/project/laya/ |
| - **Demo** https://huggingface.co/spaces/convaiinnovations/laya-demo |
| |
| Apache 2.0 · Convai Innovations |
| |
