ukr_50m

A Ukrainian-only model. It is one of the 40 checkpoints released with the paper "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations" (EMNLP 2026).

Paper (EMNLP 2026) | arXiv | Project page | Code

Model

All 40 models in the study share one architecture, tokenizer, training recipe and data source (BabyBabelLM). Each is a decoder-only model with about 310M parameters (12 layers, hidden size 768, 12 attention heads, 512-token context), trained from scratch. The tokenizer and vocabulary come from meta-llama/Llama-3.2-1B; no pretrained Llama weights are loaded.

This checkpoint:

  • Language: Ukrainian
  • Data: BabyLM-community/babylm-ukr
  • Training tokens: about 49.2M (1,500 steps of 32,768 tokens)
  • Seed: 42

In the paper the second-language-only models are used for the second-language validation, which compares them with the bilingual English and Ukrainian models.

The full list of runs, with data sources, token budgets, seeds and optimization settings, is in the training manifest of the code repository.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("iamshnoo/ukr_50m")
model = AutoModelForCausalLM.from_pretrained("iamshnoo/ukr_50m")

These are small models trained on about 50M to 100M tokens for a study of internal representations. They are meant for representation analysis and for reproducing the paper, and they are not meant for text generation in applications.

Citation

@inproceedings{mukherjee-etal-2026-double,
    title = "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations",
    author = "Mukherjee, Anjishnu  and
      Zhu, Ziwei  and
      Anastasopoulos, Antonios",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Junyi Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.2062/",
    doi = "10.18653/2026.emnlp-main.2062",
    pages = "46953--46977",
    ISBN = "979-8-89176-459-0"
}
Downloads last month
17
Safetensors
Model size
0.3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train iamshnoo/ukr_50m

Collection including iamshnoo/ukr_50m

Paper for iamshnoo/ukr_50m