TurkLM-scratch-0.3B

A 316M-parameter Turkish causal language model, trained fully from scratch (random initialization, no weight transfer from any existing checkpoint) on Turkish-only data, using the Qwen3 architecture and a 32,000-token Turkish BPE tokenizer trained specifically for this model.

Evaluation

Model Params BPC ↓ XCOPA-TR ↑ Fertility ↓ Speed (tok/s) ↑
TurkLM-scratch-0.3B 316M 1.0108 56.0% 1.80 64.9
TurkLM-scratch-0.5B 473M 0.9956 54.0% 1.80 53.5
Qwen3-0.6B 596M 1.6026 54.2% 2.69 51.4
Qwen3-1.7B 1,721M 1.3467 51.0% 2.69 37.3
Turkcell-LLM-7b-v1 7,376M 0.9528 56.2% 2.17 15.4
  • BPC = bits per character on 200 Turkish Wikipedia articles created after the training data was collected (page id ≥ 4,500,000 of the 2026-11-01 dump, roughly February 2026 onward; 958,229 characters). Lower is better, and unlike perplexity it is comparable across tokenizers. The articles are shipped in eval/data/fresh_wiki.json. (An older 200-article test set lies inside the training stream, so it is not used here.)
  • XCOPA-TR = 500 two-choice commonsense questions, random = 50%. All base models are close to chance, so this column does not separate them.
  • Fertility = tokens per Turkish word (lower means more text per token). Speed = tokens per second generating 100 tokens, bfloat16, on one Apple-silicon Mac, all models measured in the same session; compare only the ratios, not the absolute numbers. Because TurkLM needs fewer tokens per word, its speed in Turkish words per second is higher than the token column suggests (TurkLM-0.3B 36, TurkLM-0.5B 30, Qwen3-0.6B 19, Qwen3-1.7B 14, Turkcell-7B 7).
  • On this test the TurkLM models beat Qwen3-0.6B and Qwen3-1.7B on BPC. Turkcell-LLM-7b-v1, a 7B model that is 23 times larger, is better than this model, by about 6%.
  • 200 articles is a small test; differences below about 0.02 BPC should not be over-read.

All numbers can be reproduced with the scripts and data in eval/.

Model details

Parameters 316M (315.9M measured)
Architecture Qwen3-style decoder: GQA, SwiGLU, RoPE, RMSNorm, tied embeddings
Hidden size 1,024
Layers 24
Attention heads 16 (query) / 8 (KV, grouped-query attention)
FFN intermediate size 2,816
Head dimension 64
RoPE base (θ) 1,000,000
Max position embeddings 8,192 (trained only on 1,024-token sequences)
Vocabulary 32,000 (Turkish-only BPE, ByteLevel, NFC-normalized)
Precision bfloat16

Training

100,000 steps at a constant effective batch of 64 sequences × 1,024 tokens (65,536 tokens/step), about 6.5 billion tokens (a logged count), on Turkish Wikipedia (30%) and the Turkish subset of CulturaX (70%).

Tokenizer

ByteLevel BPE, 32,000 tokens, trained from scratch on 500,000 documents (70% Turkish Wikipedia, 30% a Turkish synthetic corpus) with NFC normalization. The vocabulary comes only from Turkish text, which gives much lower fertility on Turkish than general multilingual tokenizers.

Intended use

A research base model for Turkish text modelling, tokenization studies, and as a starting point for fine-tuning. It is not instruction-tuned, has had no safety training, and can produce incorrect or biased text. It was trained on far less data than multilingual models of the same size, so its world knowledge is limited: do not rely on it for factual question answering without retrieval or fine-tuning.

Known limitations

  • Base model only: no instruction tuning or RLHF.
  • Single training run, no seed variation.
  • Evaluated mainly on Wikipedia-style text; behavior on conversational or domain-specific Turkish is untested.

Training data and licenses

Turkish Wikipedia (wikimedia/wikipedia, 20231101.tr; CC BY-SA 3.0 and GFDL) and the Turkish subset of CulturaX (uonlp/CulturaX; its terms follow those of mC4 and OSCAR), mixed roughly 30% / 70% by tokens. The weights are released under Apache-2.0; if data licenses matter for your use, check the terms of these datasets yourself.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("eademir/TurkLM-scratch-0.3B")
model = AutoModelForCausalLM.from_pretrained("eademir/TurkLM-scratch-0.3B", dtype="bfloat16")

inputs = tok("Türkiye'nin başkenti", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

(On Apple GPUs use attn_implementation="eager": the default attention kernel on macOS 27 / torch 2.12.1 was not causal in our tests.)

Downloads last month
3
Safetensors
Model size
0.3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support