DilmaΓ§ 28M: English β†’ Turkish

Dilmaç (dil-maç) is an old Turkic word for an interpreter. This is a small English→Turkish translation model: 28M parameters, 112 MB in fp32 (56 MB in bf16). It scores 54.8 chrF++ on FLORES-200, runs fully offline, and is small enough for a laptop CPU.

The -28m suffix gives the size. Larger siblings, if they are made, will follow the same naming.

Direction English β†’ Turkish (one way)
Parameters 28.1M (18.9M excluding embeddings)
Architecture Encoder-decoder Transformer, 8 encoder layers + 2 decoder layers
Vocabulary 24,000-token BPE, shared by English and Turkish
Input length Sentence-level, up to 128 source tokens
License Apache-2.0

Quick start

pip install torch transformers
from transformers import AutoModel, AutoTokenizer

repo = "edededdy/dilmac-en-tr-28m"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True)

model.translate("Where is the nearest train station?", tok)
# 'En yakΔ±n tren istasyonu nerede?'

model.translate([
    "Could you send me the report by Friday, please?",
    "You look nice in the suit.",
], tok)
# ['Raporu Cuma\'ya kadar gΓΆnderir misiniz lΓΌtfen?', 'KostΓΌmΓΌn iΓ§inde gΓΌzel gΓΆrΓΌnΓΌyorsun.']

The model uses its own architecture, so loading it needs trust_remote_code=True. The code it runs is modeling_dilmac.py and configuration_dilmac.py in this repository, plain PyTorch of about 300 lines. Translation goes through model.translate(). It does not use generate() or pipeline().

Faster inference: bf16 gives the same quality (see the results below) at half the size, and is much faster on CPU:

import torch
model = model.to(torch.bfloat16)          # CPU
model = model.to("cuda", torch.bfloat16)  # GPU

Decoding options

translate(texts, tokenizer, num_beams=4, no_repeat_ngram_size=4, max_length_ratio=1.5, length_penalty=1.0, batch_size=32)

Option Default Meaning
num_beams 4 Beam width; 1 = greedy (about 3Γ— faster, about 1.5 chrF++ lower)
no_repeat_ngram_size 4 Never repeat a 4-token sequence (prevents loops)
max_length_ratio 1.5 Stop after 1.5 Γ— source length + 5 tokens
length_penalty 1.0 Beam score = log p / ((5 + len) / 6) ^ length_penalty
batch_size 32 Sentences per batch (sorted by length internally)

The defaults are the best settings we found. A beam of 5 with length penalty 0.6 was no better.

Results

sacrebleu scores on the full test sets: chrF++ (word_order=2) and BLEU (13a tokenization). Higher is better for both. For Turkish, chrF++ is the more meaningful of the two: it gives credit for partly correct word forms, while BLEU counts any wrong suffix as a miss.

Test set Sentences Greedy chrF++ / BLEU Beam 4 chrF++ / BLEU Beam 4 (bf16) chrF++ / BLEU
FLORES-200 devtest 1,012 53.29 / 24.38 54.83 / 26.32 54.75 / 26.10
WMT17 newstest2017 3,006 47.86 / 18.59 49.37 / 20.17 49.36 / 20.16
NTREX-128 1,997 44.05 / 15.58 45.48 / 17.12 45.60 / 17.10

Speed: all speeds were measured on a laptop, with English→Turkish batches of 64 sentences.

Device Greedy Beam 4
AMD Radeon 890M integrated GPU, fp32 ~145 sent/s ~46 sent/s
AMD Radeon 890M integrated GPU, bf16 ~70 sent/s
Laptop CPU, bf16 ~30 sent/s

Examples

These are beam-search outputs (default settings), including some mistakes.

English DilmaΓ§ 28M
Where is the nearest train station? En yakΔ±n tren istasyonu nerede?
Could you send me the report by Friday, please? Raporu Cuma'ya kadar gΓΆnderir misiniz lΓΌtfen?
Dr. Ehud Ur, professor of medicine at Dalhousie University in Halifax, Nova Scotia and chair of the clinical and scientific division of the Canadian Diabetes Association cautioned that the research is still in its early days. Dr. Ehud Ur, Halifax, Nova Scotia'daki Dalhousie Üniversitesi'nde tıp profesârü ve Kanada Diyabet Derneği'nin klinik ve bilimsel bâlüm başkanı, araştırmanın hala ilk günlerinde olduğuna dikkat çekti.
A 28-year-old chef who had recently moved to San Francisco was found dead in the stairwell of a local mall this week. Yakın zamanda San Francisco'ya taşınan 28 yaşındaki bir şef, bu hafta yerel bir alışveriş merkezinde âlü bulundu. (drops "in the stairwell")
I would be glad to talk to you about the new workers' brigade! Or anything else! Yeni işçilerin tugayı hakkında konuşmaktan mutluluk duyarım! (drops "Or anything else!")
28-Year-Old Chef Found Dead at San Francisco Mall San Francisco Mall Alışveriş Merkezi'de Γ–lΓΌ Bulundu San Francisco Mall Mall'da Γ–lΓΌ 28 YaşlΔ± Chef Bulun (headline: garbled)

Model details

Pre-norm Transformer encoder-decoder:

Width / heads d_model 384, 6 attention heads (head size 64)
Feed-forward SwiGLU, hidden size 1024
Layers 8 encoder, 2 decoder (a deep encoder and a shallow decoder, for fast decoding)
Normalization RMSNorm (pre-norm), plus a final norm in the encoder and in the decoder
Positions Rotary embeddings (RoPE) in self-attention; none in cross-attention
Embeddings One matrix shared by source, target and output projection
Biases None

Special tokens: <pad> = 0, <s> = 1 (decoder start), </s> = 2, <sep> = 3 (unused by this model).

Training

  • Data: about 70M English–Turkish sentence pairs from publicly available parallel corpora. Before training, the pairs were filtered by length ratio, script and language checks, sentence completeness, and template and placeholder text, then deduplicated.
  • Tokens: a single pass over 2.82B tokens (source + target), 274k steps.
  • Batches: about 12k tokens per batch, with source and target capped at 128 tokens.
  • Optimizer: AdamW (Ξ² = 0.9, 0.98). Peak learning rate 1e-3 with 400 warmup steps, then cosine decay over tokens to 10% of the peak.
  • Regularization: label smoothing 0.1, no dropout.
  • Precision: bf16 autocast.
  • Released weights: the average of the last 5 checkpoints, about 30 minutes of training apart.
  • Hardware: one AMD Radeon 890M integrated laptop GPU (ROCm on Windows), for 58.6 hours.
  • Selection: the shape was chosen in a size-matched architecture race between deep-encoder, balanced, decoder-only and wide variants.

Benchmark overlap: sentences matching FLORES-200 and NTREX-128 were removed from most of the training data. A few hundred very short, generic lines that also appear in those test sets remain (e.g. one-line greetings).

Intended use

  • Getting the gist of English text in Turkish: articles, documents, messages.
  • Translation on devices where a large model or a cloud API isn't an option.
  • Drafts for a human translator to post-edit.
  • A small, fast baseline for English–Turkish MT research and distillation experiments.

Limitations

  • English β†’ Turkish only. It will not translate Turkish into English.
  • Headlines and Title-Case text are a clear weak spot. Output may be garbled or repetitive (see the examples).
  • Sentence level only. It has no document context, so the formal siz and informal sen, pronouns and terminology may be inconsistent across sentences. Split long texts into sentences.
  • Long, complex sentences sometimes lose a clause.
  • Rare names, technical terms and numbers are less reliable than everyday vocabulary.
  • Inputs longer than 128 tokens (about 100 English words) are truncated.
  • It is much smaller than commercial systems and large language models, and less accurate on hard text. Do not use its output unreviewed where errors matter: legal, medical, safety or publication.
  • Like any model trained on web and media text, it can reproduce biases from its training data, for example gendered word choices in contexts the English leaves unspecified.

Reproducing the scores

import json, sacrebleu, torch
from transformers import AutoModel, AutoTokenizer

repo = "edededdy/dilmac-en-tr-28m"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).to("cuda")

src, ref = ...  # lists of English sources and Turkish references
hyp = model.translate(src, tok, batch_size=64)
print(sacrebleu.corpus_chrf(hyp, [ref], word_order=2).score, sacrebleu.corpus_bleu(hyp, [ref]).score)

License

Apache-2.0, for the weights and the code.

Downloads last month
29
Safetensors
Model size
28.1M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Space using edededdy/dilmac-en-tr-28m 1

Collection including edededdy/dilmac-en-tr-28m

Evaluation results