Upload BENCHMARKS.md with huggingface_hub
Browse files- BENCHMARKS.md +106 -0
BENCHMARKS.md
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# jeb benchmarks
|
| 2 |
+
|
| 3 |
+
Every model answered **byte-identical questions** in the same run (fixed seed, 400 cases per task,
|
| 4 |
+
1,600 total). Unlike comparisons that cite published figures, **every number here was measured on
|
| 5 |
+
this hardware** โ including Jev, via its live commercial API.
|
| 6 |
+
|
| 7 |
+
| run | what | where |
|
| 8 |
+
|---|---|---|
|
| 9 |
+
| jeb | RTX A4000, FP32, batch 1 | measured |
|
| 10 |
+
| Jev 1.13.0 | live API, `POST /v1/systemone` | measured, 2 independent passes |
|
| 11 |
+
| laya / laya-multilingual | same A4000, `laya==0.3.5` | measured |
|
| 12 |
+
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
## Headline โ Arabic, same tests, same prompts
|
| 16 |
+
|
| 17 |
+
| Task | **jeb** | Jev 1.13.0 | laya-multilingual | laya (English) |
|
| 18 |
+
|---|---|---|---|---|
|
| 19 |
+
| MASSIVE intent (20 options) | **0.8650** | 0.8075 | 0.3475 | 0.1250 |
|
| 20 |
+
| XNLI-ar (3 options) | **0.7600** | 0.7450 | 0.6450 | 0.4200 |
|
| 21 |
+
| SANAD topic (7 labels) | **0.9450** | 0.9325 | 0.8475 | 0.2175 |
|
| 22 |
+
| Sentiment (2 labels) | **0.9475** | 0.7475 | 0.7000 | 0.5700 |
|
| 23 |
+
| **Overall** | **0.8738** | **0.8081** | **0.6350** | **0.3331** |
|
| 24 |
+
| ECE *(lower better)* | **0.0283** | โ | 0.0810 | โ |
|
| 25 |
+
| p50 latency, 1 question | **9.3 ms** | 988 ms | 32.8 ms | โ |
|
| 26 |
+
| Cost | **$0 self-hosted** | $0.042 / Mtok | $0 | $0 |
|
| 27 |
+
|
| 28 |
+
jeb wins **every task**, at **106x lower latency** than the commercial API, for free.
|
| 29 |
+
|
| 30 |
+
### Jev reproducibility
|
| 31 |
+
|
| 32 |
+
Two independent 1,600-case passes against the live API:
|
| 33 |
+
|
| 34 |
+
| | pass 1 | pass 2 | delta |
|
| 35 |
+
|---|---|---|---|
|
| 36 |
+
| MASSIVE intent | 0.8075 | 0.8075 | 0.0000 |
|
| 37 |
+
| XNLI-ar | 0.7450 | 0.7450 | 0.0000 |
|
| 38 |
+
| topic | 0.9300 | 0.9325 | 0.0025 |
|
| 39 |
+
| sentiment | 0.7525 | 0.7475 | 0.0050 |
|
| 40 |
+
| **overall** | 0.8087 | 0.8081 | **0.0006** |
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## Held-out status
|
| 45 |
+
|
| 46 |
+
Laya's own benchmark file notes that its headline 0.766 comes from a checkpoint fine-tuned on that
|
| 47 |
+
benchmark's training split, while its base checkpoints score *below* majority-class. We label ours:
|
| 48 |
+
|
| 49 |
+
| task | held out? | note |
|
| 50 |
+
|---|---|---|
|
| 51 |
+
| MASSIVE intent | **yes** | test split; never in training |
|
| 52 |
+
| Sentiment | **yes** | test split; never in training |
|
| 53 |
+
| XNLI-ar | no | train split seen; test items unseen |
|
| 54 |
+
| SANAD topic | no | different shuffle slice |
|
| 55 |
+
|
| 56 |
+
**800 of 1,600 cases (50%) are fully held out.**
|
| 57 |
+
|
| 58 |
+
---
|
| 59 |
+
|
| 60 |
+
## On the XNLI "English bar"
|
| 61 |
+
|
| 62 |
+
The project target was to beat *English* scores with *Arabic* results. jeb clears that on intent,
|
| 63 |
+
topic, sentiment, calibration and latency. It does not clear it on NLI โ and the measurements
|
| 64 |
+
show why that bar is not an Arabic-achievable number:
|
| 65 |
+
|
| 66 |
+
| system | English XNLI | Arabic XNLI | drop |
|
| 67 |
+
|---|---|---|---|
|
| 68 |
+
| laya | **0.8825** | 0.4200 | -46 pts |
|
| 69 |
+
| laya-multilingual | 0.8430 *(published)* | 0.6450 | -20 pts |
|
| 70 |
+
| Jev 1.13.0 | *not published* | 0.7450 | โ |
|
| 71 |
+
| **jeb** | *n/a (Arabic only)* | **0.7600** | โ |
|
| 72 |
+
| CAMeLBERT *(published)* | โ | 0.557 | โ |
|
| 73 |
+
|
| 74 |
+
The same Laya architecture drops **24 points** moving from English XNLI to Arabic XNLI. XNLI-ar is
|
| 75 |
+
machine-translated from English and carries translation noise โ some items are not coherent Arabic.
|
| 76 |
+
No system measured here, commercial or open, approaches 0.88 in Arabic.
|
| 77 |
+
|
| 78 |
+
**jeb holds the highest Arabic XNLI score of every system tested.**
|
| 79 |
+
|
| 80 |
+
---
|
| 81 |
+
|
| 82 |
+
## Why Arabic-only wins
|
| 83 |
+
|
| 84 |
+
Tokens needed for identical Arabic text (lower is better):
|
| 85 |
+
|
| 86 |
+
| encoder | MSA | Gulf | Egyptian | English |
|
| 87 |
+
|---|---:|---:|---:|---:|
|
| 88 |
+
| ModernBERT-large *(laya EN)* | 55 | 59 | 54 | 19 |
|
| 89 |
+
| mmBERT-base *(laya multilingual)* | 31 | 32 | 31 | 18 |
|
| 90 |
+
| XLM-R base | 27 | 30 | 27 | 20 |
|
| 91 |
+
| AraBERTv02 | 18 | 24 | 21 | 36 |
|
| 92 |
+
| **MARBERTv2 *(jeb)*** | **17** | **22** | **17** | 26 |
|
| 93 |
+
|
| 94 |
+
ModernBERT has no Arabic vocabulary โ it falls back to raw UTF-8 bytes, producing mojibake
|
| 95 |
+
fragments. At 3.2x better compression, the same context window holds ~3x more Arabic.
|
| 96 |
+
|
| 97 |
+
---
|
| 98 |
+
|
| 99 |
+
## Reproduce
|
| 100 |
+
|
| 101 |
+
```bash
|
| 102 |
+
python build_bench_v2.py # 1,600 cases, fixed seed
|
| 103 |
+
python bench_v2_ours.py # jeb
|
| 104 |
+
python laya_ar_bench.py # both laya checkpoints
|
| 105 |
+
python jev_ar2.py # live Jev API (needs a key)
|
| 106 |
+
```
|