ytu-ce-cosmos/modernbert-tr-base

ModernBERT-TR

Release blog · Model collection

A 150M-parameter Turkish encoder with an 8,192-token context window.

Model

ModernBERT-TR is a masked-language encoder based on the ModernBERT architecture with alternating global and local attention.

Parameters	149.4M
Context length	8,192 tokens
Layers	22
Hidden width	768
Attention heads	12
GLU feed-forward width	1,152
Tokenizer	Cased WordPiece, 50,008 tokens

Results

We evaluate ModernBERT-TR with task-specific fine-tuning on 28 TabiBench tasks, the fixed five-seed TrGLUE protocol, and frozen-encoder linear probing with encoder-fast-eval.

ModernBERT-TR leads the frozen-encoder evaluation and ranks second on TabiBench and TrGLUE.

Mean rank across 47 tasks grouped by capability; lower is better. ModernBERT-TR leads classification, pairwise inference, and retrieval, and ties TabiBERT on acceptability.

Usage

from transformers import pipeline

unmask = pipeline(
    "fill-mask",
    model="ytu-ce-cosmos/modernbert-tr-base",
)
unmask("Türkiye'nin başkenti [MASK]'dır.")

Training

We pretrain from random initialization with packed sequences, dynamic masking, BF16, and StableAdamW. Our effective batch size is 1.05M tokens.

Phase	Training
1k pretraining	200B tokens at length 1,024; 6B-token warmup to 5e-4, then constant learning rate; masking 0.30
1k short decay	20B-token continuation from the 1k endpoint; decay from 5e-4 to 0; masking 0.15
1k low-LR continuation	Rollback to about 40B tokens; 100B tokens at 1e-4, masking 0.10
1k decay	1k decay soup from 1k low-LR continuation and 1k short decay
8k context extension	50B-token continuation from the 1k endpoint at length 8,192; constant 2e-4; masking 0.30
8k terminal decay	20B-token continuation on the 8k mix; decay from 2e-4 to 0; masking 0.10

The 1k pretraining mix is about 81.4% Turkish, 8.5% English, 10.0% code, and 0.1% parallel text. The 8k mix adds long-form Turkish PDF, legal, and long-document sources.

Revisions

Revision	Contents
main	ModernBERT-TR; 50% 1k decayed soup and 50% 8k terminal decay
1k-no-decay	Direct 200B-token 1k pretraining endpoint
1k-decay	50% 1k low-LR continuation and 50% 1k short decay
8k-no-decay	Direct 50B-token 8k context-extension endpoint

License

Apache-2.0.