Update RESULTS.md
Browse files- RESULTS.md +41 -0
RESULTS.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Results — laya-ara
|
| 2 |
+
|
| 3 |
+
Frozen Laya JSONL. Stock is `convaiinnovations/laya-multilingual`. Relative lift is *(model − stock) / stock*.
|
| 4 |
+
|
| 5 |
+
Card: [`README.md`](README.md) · JSON: [`results/nlu_benches.json`](results/nlu_benches.json) · all models: [`results/all_cards.json`](results/all_cards.json) · RAG: [`../laya-ara-rag/RESULTS.md`](../laya-ara-rag/RESULTS.md)
|
| 6 |
+
|
| 7 |
+
## Highlights vs base
|
| 8 |
+
|
| 9 |
+
| Arabic task | *n* | Base | laya-ara | Δ rel. |
|
| 10 |
+
|---|---:|---:|---:|---:|
|
| 11 |
+
| Intent, 20 options (MASSIVE ar-SA) | 2974 | 0.386 | **0.816** | **+111%** |
|
| 12 |
+
| Scenario, 18-way | 2974 | 0.427 | **0.865** | **+103%** |
|
| 13 |
+
| Hierarchical intent | 2694 | 0.536 | **0.893** | **+67%** |
|
| 14 |
+
| OSACT4-A macro-F1 | 1000 | 0.726 | **0.862** | **+19%** |
|
| 15 |
+
| XNLI-ar | 5010 | 0.686 | **0.723** | +5% |
|
| 16 |
+
|
| 17 |
+
## Zero-shot NLU (this mix)
|
| 18 |
+
|
| 19 |
+
| Task | *n* | Base | laya-ara |
|
| 20 |
+
|---|---:|---:|---:|
|
| 21 |
+
| OSACT4-HS (acc / F1) | 1000 | 0.932 / 0.655 | 0.875 / 0.645 |
|
| 22 |
+
| AJGT | 360 | 0.833 | 0.756 |
|
| 23 |
+
| LABR binary | 2348 | 0.759 | 0.766 |
|
| 24 |
+
| ASTD hold (acc / F1) | 1500 | 0.295 / 0.286 | 0.326 / 0.297 |
|
| 25 |
+
| TyDiQA-ar sentence pick | 298 | 0.359 / 0.155 | 0.413 / 0.170 |
|
| 26 |
+
|
| 27 |
+
## Short-list rerank (this card, no MIRACL train)
|
| 28 |
+
|
| 29 |
+
| Task | *n* | Base | laya-ara | Δ rel. |
|
| 30 |
+
|---|---:|---:|---:|---:|
|
| 31 |
+
| Mr.TyDi-ar | 2000 | 0.176 | 0.260 | +48% |
|
| 32 |
+
| SadeemQuestion | 2089 | 0.168 | 0.242 | +44% |
|
| 33 |
+
| MLQA-ar | 2000 | 0.133 | 0.214 | +61% |
|
| 34 |
+
| MIRACL-ar (dev) | 2896 | 0.153 | 0.210 | +37% |
|
| 35 |
+
| XPQA-ar | 750 | 0.213 | 0.281 | +32% |
|
| 36 |
+
| PublicHealthQA-ar | 86 | 0.116 | 0.244 | +110% |
|
| 37 |
+
| Mintaka-ar | 2203 | 0.285 | 0.311 | +9% |
|
| 38 |
+
|
| 39 |
+
Specialist rerank numbers: [`laya-ara-rag`](https://huggingface.co/Wouze/laya-ara-rag).
|
| 40 |
+
|
| 41 |
+
Contact: [Mohammad Alkhenizan](https://www.linkedin.com/in/mohammad-alkhenizan-537623257).
|