en_ukr_b

A bilingual model trained on English and Ukrainian. It is one of the 40 checkpoints released with the paper "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations" (EMNLP 2026).

Paper (EMNLP 2026) | arXiv | Project page | Code

Model

All 40 models in the study share one architecture, tokenizer, training recipe and data source (BabyBabelLM). Each is a decoder-only model with about 310M parameters (12 layers, hidden size 768, 12 attention heads, 512-token context), trained from scratch. The tokenizer and vocabulary come from meta-llama/Llama-3.2-1B; no pretrained Llama weights are loaded.

This checkpoint:

  • Languages: English and Ukrainian
  • English data: the disjoint English partition, which has no documents in common with the shared partition used by en_50m
  • Ukrainian data: BabyLM-community/babylm-ukr
  • Schedule: 50-step English/L2 blocks, alternating for 3,000 steps (1,500 steps per language)
  • Training tokens: about 98.3M (3,000 steps of 32,768 tokens)
  • Seed: 42

In the paper this is the bilingual model for the disjoint English documents comparison (C2, against en_50m) and the matched compute plus disjoint documents comparison (C4, against en_100m).

The full list of runs, with data sources, token budgets, seeds and optimization settings, is in the training manifest of the code repository.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("iamshnoo/en_ukr_b")
model = AutoModelForCausalLM.from_pretrained("iamshnoo/en_ukr_b")

These are small models trained on about 50M to 100M tokens for a study of internal representations. They are meant for representation analysis and for reproducing the paper, and they are not meant for text generation in applications.

Citation

@inproceedings{mukherjee-etal-2026-double,
    title = "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations",
    author = "Mukherjee, Anjishnu  and
      Zhu, Ziwei  and
      Anastasopoulos, Antonios",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Junyi Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.2062/",
    doi = "10.18653/2026.emnlp-main.2062",
    pages = "46953--46977",
    ISBN = "979-8-89176-459-0"
}
Downloads last month
16
Safetensors
Model size
0.3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train iamshnoo/en_ukr_b

Collection including iamshnoo/en_ukr_b

Paper for iamshnoo/en_ukr_b