Bichura

Bichura is a 2.4B-parameter base language model for Tatar. It was built from the Turkish model Kumru-2B-Base by replacing its vocabulary with a Tatar tokenizer and continuing pretraining on Tatar text. It is not instruction-tuned.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ilchats/Bichura")
model = AutoModelForCausalLM.from_pretrained("ilchats/Bichura", dtype="bfloat16")

inputs = tokenizer("Казан шәһәре", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model

Mistral-style decoder with 18 layers, hidden size 3072, 16 attention heads and 4 key-value heads. Input and output embeddings are untied. The tokenizer is the 48K SentencePiece Unigram tokenizer of TATlit. Continued pretraining used 4096-token sequences. Weights are stored in bf16.

Training

1. Vocabulary transfer. The Kumru tokenizer was replaced with the TATlit tokenizer. The input and output embedding of each new token was initialized as a weighted average of up to 20 Kumru token embeddings. The weights come from a frequency table of Turkish word forms paired with their Tatar Cyrillic renderings.
2. Embedding warm-up. With the transformer layers frozen, the input and output embeddings were trained on 16M tokens of Turkish text rendered in Tatar Cyrillic.
3. Continued pretraining. All weights were trained on 2.23B tokens with a warmup-stable-decay schedule. Peak learning rate 5e-5, 84 warmup steps, constant rate until step 3782, then 476 steps of cosine decay to 5e-6. Batch size 128 × 4096 tokens, AdamW, weight decay 0.1. In the stable phase 7% of the training sequences were Turkish text rendered in Tatar Cyrillic. The decay phase used a 250M-token high-quality subset of the Tatar corpus with normalized spelling.

Training took about 30 hours on one H100 80GB.

Results

Bits per byte on public sets, lower is better. FLORES-200 dev and devtest (2009 sentences), BOUQuET (1358 sentences) and UD Tatar-NMCTT (148 sentences). Each sentence is scored separately.

Model	Params	FLORES	BOUQuET	UD
Bichura	2.4B	.722	.741	.489
TATlit	478M	.740	.777	.502
Tweety-7B	7B	.757	.800	.544
Gemma-4-31B	31B	.772	.939	.691
Goldfish-tat	125M	.778	.801	.502
mGPT-1.3B	1.3B	.950	1.186	.880
gpt-oss-120b	120B	1.092	1.164	1.099
Qwen3-32B	32B	1.144	1.423	1.159
Llama-4-Scout	109B	1.222	1.498	1.080

TatBLiMP, 1248 minimal pairs in 16 grammatical phenomena, higher is better.

Model	Params	acc	acc_norm
Bichura	2.4B	.984	.973
TATlit	478M	.975	.958
Goldfish-tat	125M	.974	.958
Tweety-7B	7B	.956	.915
Gemma-4-31B	31B	.924	.839
Llama-4-Scout	109B	.889	.806
Qwen3-32B	32B	.811	.708
gpt-oss-120b	120B	.803	.677
mGPT-1.3B	1.3B	.736	.639

Numbers for other models in the two tables above are from the TATlit model card, measured with the same harness.

TUNSEV, Turkic Unified Next-sentence Evaluation. The benchmark measures how well a model captures the connection between adjacent sentences, distinguishing the true continuation of a text from a sentence taken from elsewhere. Parallel tasks make it possible to compare models across languages. Higher is better.

Model	Params	FLORES	BOUQuET	Literature	All
Kumru-2B-Base, Turkish tasks	2.4B	88.6	83.0	81.5	82.9
Bichura	2.4B	86.7	81.3	81.0	82.1
Tweety-7B	7B	87.5	81.3	77.5	79.8
TATlit	478M	85.1	77.9	76.2	78.0
Goldfish-tat	125M	83.3	75.3	72.0	74.4
mGPT-1.3B	1.3B	78.6	66.9	63.3	66.5

Kumru-2B-Base is scored on the parallel Turkish versions of the same tasks. The literature part is in the training data of Bichura and TATlit, and most likely of Kumru-2B-Base. For the other models this is not known.

Limitations

Base model with no instruction following. Trained on Tatar in Cyrillic script. The lowest TatBLiMP scores are island effects (acc .887) and numeral number (acc_norm .882).

Data

The Tatar corpus has 1.88B tokens in 3.94M documents from books, periodicals, news sites, web crawls, social media, forums, Wikipedia and legal texts. The Turkish part is 2159 books rendered in Tatar Cyrillic. The training corpus is not released.

Authors

Ilshat Saetov, Oleg Sokolov.

License

Weights and tokenizer are released under CC BY-NC-SA 4.0. Kumru-2B-Base is released under Apache 2.0.

Citation

@misc{bichura2026,
  title        = {{Bichura}: a base language model for Tatar},
  author       = {Saetov, Ilshat and Sokolov, Oleg},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ilchats/Bichura}},
  note         = {Contributors: Dmitry Gaynullin, Rinat Kharisov, Raynur Khasanov, Ilgiz Jihangir, Adelia Khasanova}
}