TurkLM-scratch-0.5B

A 473M-parameter Turkish causal language model, trained fully from scratch (random initialization, no weight transfer from any existing checkpoint) on Turkish-only data, using the Qwen3 architecture and the same 32,000-token Turkish BPE tokenizer as TurkLM-scratch-0.3B.

Evaluation

Model Params BPC ↓ XCOPA-TR ↑ Fertility ↓ Speed (tok/s) ↑
TurkLM-scratch-0.3B 316M 1.0108 56.0% 1.80 64.9
TurkLM-scratch-0.5B 473M 0.9956 54.0% 1.80 53.5
Qwen3-0.6B 596M 1.6026 54.2% 2.69 51.4
Qwen3-1.7B 1,721M 1.3467 51.0% 2.69 37.3
Turkcell-LLM-7b-v1 7,376M 0.9528 56.2% 2.17 15.4
  • BPC = bits per character on 200 Turkish Wikipedia articles created after the training data was collected (page id ≥ 4,500,000 of the 2026-11-01 dump, roughly February 2026 onward; 958,229 characters). Lower is better, and unlike perplexity it is comparable across tokenizers. The articles are shipped in eval/data/fresh_wiki.json. (An older 200-article test set lies inside the training stream, so it is not used here.)
  • XCOPA-TR = 500 two-choice commonsense questions, random = 50%. All base models are close to chance, so this column does not separate them.
  • Fertility = tokens per Turkish word (lower means more text per token). Speed = tokens per second generating 100 tokens, bfloat16, on one Apple-silicon Mac, all models measured in the same session; compare only the ratios, not the absolute numbers. Because TurkLM needs fewer tokens per word, its speed in Turkish words per second is higher than the token column suggests (TurkLM-0.3B 36, TurkLM-0.5B 30, Qwen3-0.6B 19, Qwen3-1.7B 14, Turkcell-7B 7).
  • On this test the TurkLM models beat Qwen3-0.6B and Qwen3-1.7B on BPC. Turkcell-LLM-7b-v1, a 7B model that is 15 times larger, is better, by 4% over the 0.5B model.
  • 200 articles is a small test; differences below about 0.02 BPC should not be over-read.

All numbers can be reproduced with the scripts and data in eval/.

Model details

Parameters 473M (473.2M measured)
Architecture Qwen3-style decoder: GQA, SwiGLU, RoPE, RMSNorm, tied embeddings
Hidden size 1,024
Layers 28
Attention heads 16 (query) / 8 (KV, grouped-query attention)
FFN intermediate size 3,072
Head dimension 128
RoPE base (θ) 10,000
Max position embeddings 40,960 (trained only on 1,024-token sequences)
Vocabulary 32,000 (Turkish-only BPE, ByteLevel, NFC-normalized)
Precision bfloat16

Note on the RoPE base. This model's RoPE base is 10,000 (the 0.3B model uses 1,000,000). The training script for this model builds its config without setting rope_theta, so the library's default applied. We checked which value the trained weights expect: on 60 fresh articles the model gets 1.034 bits per character with 10,000 and 1.277 with 1,000,000. Keep rope_theta as saved in config.json; changing it makes the model clearly worse. Because of this, the two TurkLM models are not a perfectly controlled size comparison.

Training

142,900 optimizer steps in several phases (batch size changed during training): about the first 40,800 steps at 32,768 tokens/step, then 131,072 tokens/step until step 140,000, and 196,608 tokens/step for the last 2,900 steps, about 14.9 billion tokens in total. The token count is the step count times each phase's batch size, reconstructed from the configuration changes, not a logged count (uncertainty about ±1B). Data: Turkish Wikipedia (30%) and Turkish CulturaX (70%); the interleaving settings differed slightly between phases. This is the final checkpoint; earlier checkpoints are not released (an earlier 100,000-step checkpoint scored 0.9931 on the same BPC test, i.e. the same within noise).

Tokenizer

ByteLevel BPE, 32,000 tokens, trained from scratch on 500,000 Turkish documents with NFC normalization.

Intended use

A research base model for Turkish text modelling, tokenization studies, and as a starting point for fine-tuning. It is not instruction-tuned, has had no safety training, and can produce incorrect or biased text. It was trained on far less data than multilingual models of the same size, so its world knowledge is limited: do not rely on it for factual question answering without retrieval or fine-tuning.

Known limitations

  • Base model only: no instruction tuning or RLHF.
  • Single training run, no seed variation.
  • Evaluated mainly on Wikipedia-style text; behavior on conversational or domain-specific Turkish is untested.

Training data and licenses

Turkish Wikipedia (wikimedia/wikipedia, 20231101.tr; CC BY-SA 3.0 and GFDL) and the Turkish subset of CulturaX (uonlp/CulturaX; its terms follow those of mC4 and OSCAR), mixed roughly 30% / 70% by tokens. The weights are released under Apache-2.0; if data licenses matter for your use, check the terms of these datasets yourself.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("eademir/TurkLM-scratch-0.5B")
model = AutoModelForCausalLM.from_pretrained("eademir/TurkLM-scratch-0.5B", dtype="bfloat16")

inputs = tok("Türkiye'nin başkenti", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

(On Apple GPUs use attn_implementation="eager": the default attention kernel on macOS 27 / torch 2.12.1 was not causal in our tests.)

Downloads last month
3
Safetensors
Model size
0.5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support