hackoffice convaiinnovations commited on
Commit
b525d41
·
0 Parent(s):

Duplicate from convaiinnovations/laya-multilingual

Browse files

Co-authored-by: Convai Innovations <convaiinnovations@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,181 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-classification
5
+ language: [multilingual, en, de, fr, es, pt, it, nl, sv, da, nb, ru, pl, tr, ar, he, fa, ur, hi, bn, ta, te, kn, ml, th, vi, id, ms, tl, ja, ko, zh, el, hu, fi, ro, sq, sl, sw, af, cy, am, hy, ka, km, my, mn, lv, is, az, jv]
6
+ tags: [laya, multilingual, mmbert, system-one, calibrated-decisions, rlcd, classification, routing, guardrails, moderation, commercial-use]
7
+ ---
8
+
9
+ <p align="center">
10
+ <img src="https://huggingface.co/convaiinnovations/laya/resolve/main/assets/logo-mark.png" alt="" width="72" />
11
+ </p>
12
+
13
+ # Laya Multilingual
14
+
15
+ Non-autoregressive **System 1 decision model** covering 100+ languages. Give it a **state**
16
+ (text, email, ticket, or JSON) and **typed questions**; it returns typed answers with
17
+ probabilities in a single forward pass. No text generation, so nothing to parse and nothing to
18
+ hallucinate.
19
+
20
+ Part of the [Laya family](https://huggingface.co/convaiinnovations/laya) — **use this checkpoint
21
+ for anything that is not English.**
22
+
23
+ | checkpoint | encoder | params | context | use it for |
24
+ |---|---|---|---|---|
25
+ | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) | ModernBERT-large | 421M | 512 | English |
26
+ | **`convaiinnovations/laya-multilingual`** (this repo) | mmBERT-base | 322M | 1024 (up to 8,192) | 100+ languages, ~2x faster |
27
+ | [`convaiinnovations/laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions) | ModernBERT-large | 421M | 1024 | the typed-decisions workflows |
28
+
29
+ > **Long documents: `laya-multilingual` reads up to 8,192 tokens.** It ships with a 1,024-token limit that cuts long documents off, so pass `max_len=8192` for them:
30
+ >
31
+ > ```python
32
+ > import laya
33
+ > agent = laya.load("convaiinnovations/laya-multilingual")
34
+ > result = agent.predict(long_document, questions, max_len=8192)
35
+ > ```
36
+ >
37
+ > In the table below, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary (8 to 17 of 20), so check long-document accuracy on your own data. Short inputs give identical answers with `max_len=8192`, and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU.
38
+
39
+ <p align="center">
40
+ <img src="https://raw.githubusercontent.com/NandhaKishorM/laya/main/assets/long_context_8192.png" alt="laya-multilingual long-document accuracy by document length" width="100%" />
41
+ </p>
42
+
43
+ ## Quickstart
44
+
45
+ ```bash
46
+ pip install laya
47
+ ```
48
+
49
+ ```python
50
+ import laya
51
+
52
+ agent = laya.load("convaiinnovations/laya-multilingual")
53
+ result = agent.predict(
54
+ {"body": "मुझसे इनवॉइस 4411 के लिए दो बार शुल्क लिया गया। कृपया आज ही धनवापसी करें।"},
55
+ {"department": {"type": "choice", "instructions": "Which team should handle `body`?",
56
+ "criteria": {"billing": "invoices, payments, refunds",
57
+ "technical": "bugs and outages", "sales": "pricing"}},
58
+ "refund_requested": {"type": "noul", "instructions": "Does the sender ask for money back?"}},
59
+ )
60
+ print(result["answers"]["department"]["choice"]) # billing
61
+ ```
62
+
63
+ ### Let the Router choose
64
+
65
+ ```python
66
+ from laya import Router
67
+
68
+ router = Router()
69
+ router.predict({"body": "I was charged twice"}, questions) # -> laya
70
+ router.predict({"body": "二重に請求されました"}, questions) # -> laya-multilingual
71
+ ```
72
+
73
+ The default `Router()` keeps both `english` and this checkpoint resident, so a
74
+ mixed workload no longer swaps checkpoints on every language change. For a server, load them up
75
+ front so even the first request of each language is just a forward pass:
76
+
77
+ ```python
78
+ router = Router()
79
+ router.preload(["english", "multilingual"]) # both resident; no swap at request time
80
+ ```
81
+
82
+ `router.attach("multilingual", agent)` registers an `Agent` you already built, so a process that
83
+ loaded this checkpoint directly can hand it to the router instead of loading it twice.
84
+
85
+ More text reaches this checkpoint than script alone would send: plain-ASCII Spanish, Italian, Portuguese and French
86
+ (accents stripped by mail clients and ticket systems), Brazilian Portuguese support text, and any
87
+ script the router has no range for. If you already run a language-identification model, pass its
88
+ answer with `router.predict(state, questions, lang_guess=code_or_callable)`.
89
+
90
+ It also receives CJK requests that contain Latin brand names, romanized
91
+ Bangla, and Azerbaijani. On 20,000 English texts, at most 5 English sentences move, all quoting long
92
+ native-script names.
93
+
94
+ Routing is decided from the script of the input, **before** the forward pass — because the
95
+ model's confidence gives no warning when a checkpoint cannot read its input (see below).
96
+
97
+ > **If `laya.load()` hangs:** `transformers` probes for TensorFlow at import, and when TF is
98
+ > installed its abseil runtime can deadlock model construction. Run with `USE_TF=0`.
99
+
100
+ ## Why this checkpoint exists
101
+
102
+ Measured across **all 51 MASSIVE languages**, intent classification with 20 options
103
+ (random = 0.050), both checkpoints answering byte-identical questions:
104
+
105
+ | | `laya` (English) | **`laya-multilingual`** |
106
+ |---|---|---|
107
+ | macro accuracy | 0.227 | **0.366** |
108
+ | macro ECE | 0.733 | **0.387** |
109
+ | languages clearing 3x random | 23 / 51 | **45 / 51** |
110
+
111
+ The English checkpoint does not degrade gracefully outside English — it collapses, and stays
112
+ confident while doing so. Khmer: **0.000 accuracy at 0.952 confidence**. Hebrew 0.060, Armenian
113
+ 0.050 (exactly random), Bengali 0.080 — all reported with 0.89–0.96 confidence. Its mean
114
+ confidence never drops below 0.885 at any accuracy level, so confidence gating cannot catch it.
115
+
116
+ Per-language, this checkpoint turns near-random into usable: Arabic 0.110 → **0.400**, Bengali
117
+ 0.080 → **0.290**, Azerbaijani 0.100 → **0.300**, Hindi 0.100 → **0.387**, Korean 0.110 →
118
+ **0.490**, Turkish 0.140 → **0.437**.
119
+
120
+ ### XNLI (15 languages)
121
+
122
+ | | `laya` | **`laya-multilingual`** |
123
+ |---|---|---|
124
+ | English | **0.860** | 0.843 |
125
+ | 14 other languages | 0.521 | **0.731** |
126
+
127
+ ### Speed — it is also the faster checkpoint
128
+
129
+ | questions per call | `laya` | **`laya-multilingual`** |
130
+ |---|---|---|
131
+ | 1 | 39.5 ms | **32.8 ms** |
132
+ | 10 | 158.6 ms (15.9 ms/q) | **72.3 ms (7.2 ms/q)** |
133
+ | 50 | 771 ms | **337 ms (6.8 ms/q)** |
134
+
135
+ 103–332 questions/sec batched on one T4, despite a 256k vocabulary — the 768-dim / 22-layer
136
+ encoder is cheaper per token than 1024-dim / 28-layer, and the gap widens with batch size.
137
+
138
+ ## Architecture
139
+
140
+ - **Backbone** mmBERT-base (307M, bidirectional, 22 layers, hidden 768, 256k vocab) + a decision
141
+ head trained from scratch: 2 transformer layers, an option-marker scorer, and an act/escalate
142
+ head. 322M total.
143
+ - **Option markers** every option is scored at its own `[MASK]` token, then softmaxed over that
144
+ question's options — so the answer space is defined per request, with no retraining.
145
+ - **Budget** 1024 tokens per question, of which 256 go to the question and its options.
146
+ - Trained from scratch with RLCD: 15,987 updates, 4 epochs, ~4.97 h.
147
+
148
+ ## Limits
149
+
150
+ - **Ships uncalibrated.** `temperature = [1.0, 1.0, 1.0]` with no per-option-count buckets. It is
151
+ systematically over-confident (mean confidence 0.75–0.83 against much lower accuracy). Refitting
152
+ one temperature per (question type, option count) on held-out data moves mean ECE
153
+ **0.314 → 0.106**. Do this on your own data before trusting the probabilities.
154
+ - **Weaker on English** than the English checkpoint: 0.619 vs 0.684 macro across English suites.
155
+ Route rather than replace.
156
+ - **Near chance on typed-decisions zero-shot** — 0.342, against a 0.318 random and 0.461
157
+ majority-class baseline. Fine-tune for a specific workflow; that is where the capability comes
158
+ from.
159
+ - **Keep `choice` questions under ~20 options.** Options share the fixed 256-token head budget, so
160
+ a very large label space leaves only a few tokens per label and accuracy falls off sharply.
161
+ - Low-resource languages are weak, not fixed: Swahili 0.210, Tamil 0.250, Amharic 0.110.
162
+ - Ordinal `score` questions are the weakest primitive (SST-5 0.282), and this checkpoint has a
163
+ **measured position bias on them**: it rarely picks the first-listed level, in any language,
164
+ including English (0 of 290 in one independent run,
165
+ [#131](https://github.com/NandhaKishorM/laya/issues/131)). For English score questions use the
166
+ English checkpoint. For other languages, validate score outputs on your own data first.
167
+ - **`noul` can under-report "true" here.** On a clearly positive input, one measurement put
168
+ `P(true)` at about 0.5 while the negative case was correctly near 0
169
+ ([#156](https://github.com/NandhaKishorM/laya/issues/156)). If `noul` answers look weak, the same
170
+ question as a two-option `choice` with neutral keys (`A` / `B`) and yes/no descriptions is a
171
+ useful check.
172
+
173
+ ## Links
174
+
175
+ - **Docs** https://nandhakishorm.github.io/laya/
176
+ - **Hub / family** https://huggingface.co/convaiinnovations/laya
177
+ - **GitHub** https://github.com/NandhaKishorM/laya · full benchmark data on the `research` branch
178
+ - **PyPI** https://pypi.org/project/laya/
179
+ - **Demo** https://huggingface.co/spaces/convaiinnovations/laya-demo
180
+
181
+ Apache 2.0 · Convai Innovations
encoder/config.json ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ModernBertForMaskedLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 2,
8
+ "classifier_activation": "gelu",
9
+ "classifier_bias": false,
10
+ "classifier_dropout": 0.0,
11
+ "classifier_pooling": "mean",
12
+ "cls_token_id": 1,
13
+ "decoder_bias": true,
14
+ "deterministic_flash_attn": false,
15
+ "dtype": "float32",
16
+ "embedding_dropout": 0.0,
17
+ "eos_token_id": 1,
18
+ "global_attn_every_n_layers": 3,
19
+ "gradient_checkpointing": false,
20
+ "hidden_activation": "gelu",
21
+ "hidden_size": 768,
22
+ "initializer_cutoff_factor": 2.0,
23
+ "initializer_range": 0.02,
24
+ "intermediate_size": 1152,
25
+ "layer_norm_eps": 1e-05,
26
+ "layer_types": [
27
+ "full_attention",
28
+ "sliding_attention",
29
+ "sliding_attention",
30
+ "full_attention",
31
+ "sliding_attention",
32
+ "sliding_attention",
33
+ "full_attention",
34
+ "sliding_attention",
35
+ "sliding_attention",
36
+ "full_attention",
37
+ "sliding_attention",
38
+ "sliding_attention",
39
+ "full_attention",
40
+ "sliding_attention",
41
+ "sliding_attention",
42
+ "full_attention",
43
+ "sliding_attention",
44
+ "sliding_attention",
45
+ "full_attention",
46
+ "sliding_attention",
47
+ "sliding_attention",
48
+ "full_attention"
49
+ ],
50
+ "local_attention": 128,
51
+ "mask_token_id": 4,
52
+ "max_position_embeddings": 8192,
53
+ "mlp_bias": false,
54
+ "mlp_dropout": 0.0,
55
+ "model_type": "modernbert",
56
+ "norm_bias": false,
57
+ "norm_eps": 1e-05,
58
+ "num_attention_heads": 12,
59
+ "num_hidden_layers": 22,
60
+ "pad_token_id": 0,
61
+ "position_embedding_type": "sans_pos",
62
+ "repad_logits_with_grad": false,
63
+ "rope_parameters": {
64
+ "full_attention": {
65
+ "rope_theta": 160000,
66
+ "rope_type": "default"
67
+ },
68
+ "sliding_attention": {
69
+ "rope_theta": 160000,
70
+ "rope_type": "default"
71
+ }
72
+ },
73
+ "sep_token_id": 1,
74
+ "sparse_pred_ignore_index": -100,
75
+ "sparse_prediction": false,
76
+ "tie_word_embeddings": true,
77
+ "transformers_version": "5.0.0",
78
+ "vocab_size": 256000
79
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204
3
+ size 643835514
rl_agent_config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "encoder": "jhu-clsp/mmBERT-base",
3
+ "head_layers": 2,
4
+ "max_len": 1024,
5
+ "head_max_len": 256,
6
+ "max_prefixes": 6,
7
+ "act_costs": {
8
+ "escalate": 0.5
9
+ },
10
+ "cost_wrong_act": 3.0,
11
+ "amp_dtype": "bf16",
12
+ "model_name": "rl-agent",
13
+ "temperature": [
14
+ 1.0,
15
+ 1.0,
16
+ 1.0
17
+ ],
18
+ "temperature_by_options": {},
19
+ "training": {
20
+ "updates": 15987,
21
+ "epochs_completed": 4,
22
+ "hours": 4.97,
23
+ "world_size": 1,
24
+ "fine_tuned_from_checkpoint": false
25
+ }
26
+ }
tokenizer/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
3
+ size 34363188
tokenizer/tokenizer_config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<bos>",
3
+ "clean_up_tokenization_spaces": false,
4
+ "cls_token": "<bos>",
5
+ "eos_token": "<eos>",
6
+ "extra_special_tokens": [
7
+ "<start_of_turn>",
8
+ "<end_of_turn>"
9
+ ],
10
+ "mask_token": "<mask>",
11
+ "model_input_names": [
12
+ "input_ids",
13
+ "attention_mask"
14
+ ],
15
+ "model_max_length": 8192,
16
+ "pad_token": "<pad>",
17
+ "padding_side": "right",
18
+ "sep_token": "<eos>",
19
+ "spaces_between_special_tokens": false,
20
+ "tokenizer_class": "PreTrainedTokenizerFast",
21
+ "unk_token": "<unk>"
22
+ }