en_50m

An English-only model. It is one of the 40 checkpoints released with the paper "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations" (EMNLP 2026).

Paper (EMNLP 2026) | arXiv | Project page | Code

Model

All 40 models in the study share one architecture, tokenizer, training recipe and data source (BabyBabelLM). Each is a decoder-only model with about 310M parameters (12 layers, hidden size 768, 12 attention heads, 512-token context), trained from scratch. The tokenizer and vocabulary come from meta-llama/Llama-3.2-1B; no pretrained Llama weights are loaded.

This checkpoint:

  • Language: English
  • English data: the shared English partition of BabyLM-community/babylm-eng
  • Training tokens: about 49.2M (1,500 steps of 32,768 tokens)
  • Seed: 42

In the paper this is the English-only reference for the matched English comparisons (C1 and C2), where it sees the same amount of English as the bilingual models.

The full list of runs, with data sources, token budgets, seeds and optimization settings, is in the training manifest of the code repository.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("iamshnoo/en_50m")
model = AutoModelForCausalLM.from_pretrained("iamshnoo/en_50m")

These are small models trained on about 50M to 100M tokens for a study of internal representations. They are meant for representation analysis and for reproducing the paper, and they are not meant for text generation in applications.

Citation

@inproceedings{mukherjee-etal-2026-double,
    title = "Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations",
    author = "Mukherjee, Anjishnu  and
      Zhu, Ziwei  and
      Anastasopoulos, Antonios",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Junyi Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.2062/",
    doi = "10.18653/2026.emnlp-main.2062",
    pages = "46953--46977",
    ISBN = "979-8-89176-459-0"
}
Downloads last month
21
Safetensors
Model size
0.3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train iamshnoo/en_50m

Collection including iamshnoo/en_50m

Paper for iamshnoo/en_50m