eademir's picture
Model card, config, tokenizer and evaluation bundle
f354976 verified
|
Raw History Blame Contribute Delete
2.82 kB

Reproducing the TurkLM evaluation

Everything needed to re-run the numbers in the model cards of TurkLM-scratch-0.3B and TurkLM-scratch-0.5B. Run it from this folder.

pip install -r requirements.txt
bash run_all.sh ..                              # this model (the folder above eval/)
bash run_all.sh Qwen/Qwen3-0.6B                 # or any Hugging Face id, for comparison
file what it does
bpc_two_sets.py bits-per-character on Turkish Wikipedia articles; main metric
data/fresh_wiki.json the 200 fresh articles (created after the training data); page ids, titles and URLs for attribution
data/bpc_docs.json the older 200-article set, inside the training stream (not held out); kept only for transparency
build_fresh_from_dump.py how the fresh set was selected (see below)
eval_xcopa.py, data/xcopa_tr_test.jsonl XCOPA-TR (500 two-choice commonsense questions, random = 50%)
eval_efficiency.py tokenizer fertility (tokens per word) and generation speed
compare_paired.py exact McNemar test between two per-item result files
results/ the per-model BPC results behind the card tables

Metric

BPC = total negative log-likelihood in bits / number of characters. Unlike perplexity it is comparable across tokenizers. Documents are cut into windows of 1,024 tokens; a window needs at least 2 tokens.

The fresh set

The models were trained on the 20231101 Turkish Wikipedia dump (highest page id 3,611,637). MediaWiki page ids grow with creation time, so every page with a larger id was created later. build_fresh_from_dump.py takes the last part of the 2026-11-01 dump (page ids 4.5M and up, roughly Feb 2026 onward, after the knowledge cutoffs of the baseline models too) and picks 200 articles (>= 1,500 characters, not redirects) evenly across the id range. The dump is the one from https://dumps.wikimedia.org/trwiki/20261101/ (part ...pages-articles1.xml-p4500001p4803437.bz2); old dumps are removed after a few months, which is why the selected articles are shipped in data/fresh_wiki.json. Article text: Wikipedia contributors, CC BY-SA 4.0; each record keeps its title and URL.

Known pitfalls

  • Apple GPU (MPS): the default sdpa attention on macOS 27 / torch 2.12.1 was not causal and gave far too low BPC. All scripts load models with attn_implementation="eager", which matches CPU and CUDA. If you write your own loader, do the same, or check against CPU.
  • Speed numbers depend on hardware and on what else the machine is doing; compare models only inside one session.
  • bfloat16: results use bf16 weights. Tiny differences (about the 4th decimal) between hardware are normal.
  • The 200-article test is small. BPC differences below about 0.02 should not be over-read.