# Reproducing the TurkLM evaluation Everything needed to re-run the numbers in the model cards of `TurkLM-scratch-0.3B` and `TurkLM-scratch-0.5B`. Run it from this folder. ```bash pip install -r requirements.txt bash run_all.sh .. # this model (the folder above eval/) bash run_all.sh Qwen/Qwen3-0.6B # or any Hugging Face id, for comparison ``` | file | what it does | |---|---| | `bpc_two_sets.py` | bits-per-character on Turkish Wikipedia articles; main metric | | `data/fresh_wiki.json` | the 200 **fresh** articles (created after the training data); page ids, titles and URLs for attribution | | `data/bpc_docs.json` | the older 200-article set, **inside the training stream** (not held out); kept only for transparency | | `build_fresh_from_dump.py` | how the fresh set was selected (see below) | | `eval_xcopa.py`, `data/xcopa_tr_test.jsonl` | XCOPA-TR (500 two-choice commonsense questions, random = 50%) | | `eval_efficiency.py` | tokenizer fertility (tokens per word) and generation speed | | `compare_paired.py` | exact McNemar test between two per-item result files | | `results/` | the per-model BPC results behind the card tables | ## Metric BPC = total negative log-likelihood in bits / number of characters. Unlike perplexity it is comparable across tokenizers. Documents are cut into windows of 1,024 tokens; a window needs at least 2 tokens. ## The fresh set The models were trained on the 20231101 Turkish Wikipedia dump (highest page id 3,611,637). MediaWiki page ids grow with creation time, so every page with a larger id was created later. `build_fresh_from_dump.py` takes the last part of the 2026-11-01 dump (page ids 4.5M and up, roughly Feb 2026 onward, after the knowledge cutoffs of the baseline models too) and picks 200 articles (>= 1,500 characters, not redirects) evenly across the id range. The dump is the one from `https://dumps.wikimedia.org/trwiki/20261101/` (part `...pages-articles1.xml-p4500001p4803437.bz2`); old dumps are removed after a few months, which is why the selected articles are shipped in `data/fresh_wiki.json`. Article text: Wikipedia contributors, CC BY-SA 4.0; each record keeps its title and URL. ## Known pitfalls * **Apple GPU (MPS):** the default `sdpa` attention on macOS 27 / torch 2.12.1 was not causal and gave far too low BPC. All scripts load models with `attn_implementation="eager"`, which matches CPU and CUDA. If you write your own loader, do the same, or check against CPU. * **Speed numbers depend on hardware and on what else the machine is doing**; compare models only inside one session. * **bfloat16:** results use bf16 weights. Tiny differences (about the 4th decimal) between hardware are normal. * The 200-article test is small. BPC differences below about 0.02 should not be over-read.