File size: 1,500 Bytes
ea80f98 d1d3fe6 ea80f98 d1d3fe6 ea80f98 d1d3fe6 ea80f98 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | # Benchmark reproduction
The benchmark compares SamAI-Tokenizer with Arabic-focused and recent multilingual
tokenizers on four datasets. It downloads tokenizer assets only, not model
weights.
## Run
From the repository root:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -r benchmark/requirements.txt
python benchmark/benchmark.py
```
Generated files are written to `results/generated/`. Override paths when
needed:
```bash
SAMAI_TOKENIZER_PATH=/path/to/tokenizer.json \
SAMAI_BENCHMARK_OUTPUT=/path/to/results \
python benchmark/benchmark.py
```
Third-party remote code is disabled by default. Some custom tokenizers will be
skipped unless you review their repositories and explicitly opt in:
```bash
SAMAI_TRUST_REMOTE_CODE=1 python benchmark/benchmark.py
```
Gated repositories also require accepting their terms and authenticating with
Hugging Face before running the script.
## Published files
- `arabic_4bench_macro_average.csv`: equal 25% weight per benchmark; primary
ranking.
- `arabic_4bench_pooled_overall.csv`: all tokens and words pooled together.
- `arabic_4bench_by_benchmark.csv`: result for every tokenizer and benchmark.
- `arabic_4bench_by_category.csv`: detailed subject, dialect, and intent
breakdowns.
- `arabicmmlu_summary.csv` and `arabicmmlu_by_subject.csv`: the original
ArabicMMLU-only comparison.
The script also produces document-level results locally. Those results are not
published because they contain copies of benchmark text.
|