r-three-moto/moto-1b-script

MoTo 1B (script-level tokenizers)

A 1B-scale Llama-style model trained from scratch on 21 languages with a mixture of tokenizers grouped by writing script; routing is described in moto_config.json.

This is a MoTo model. It has one embedding table and 11 tokenizers; during training each document was encoded by the tokenizer of its language with probability 0.9 (otherwise by one drawn uniformly).

Part of the MoTo release: paper · code · all models.

About MoTo

MoTo (Mixture of Tokenizers) replaces the single tokenizer of a multilingual language model with one tokenizer per language. Per-language BPE tokenizers are merged into a superset vocabulary that indexes one shared embedding table, and each input is encoded by the tokenizer of its language. The rest of the architecture is unchanged. The release includes MoTo models and monolithic-tokenizer baselines trained on the same 21-language data, so comparisons between them isolate the effect of tokenization.

Model details

Architecture	Llama-style decoder, 25 layers, d_model 2048, 16 heads, untied input/output embeddings
Parameters	1.96B total, 1.28B non-embedding
Tokenizer	11 tokenizers sharing a superset vocabulary (165022 embedding rows)
Context length	4096 tokens
Training	100000 steps × 1.05M tokens/step ≈ 104.86B tokens (4 sequences of 4096 tokens per GPU × 8 GPUs × 8 accumulation steps)
Data	FineWeb-Edu (English, 40% of the mixture) and FineWeb2-HQ (20 languages); exact weights in training/config.yaml
Weights	bfloat16

This is a base model trained from scratch for a controlled tokenizer comparison. It is not instruction-tuned and has no safety tuning.

Usage

The repo ships one tokenizer per language. tokenizer.py combines them into a single tokenizer that switches language:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "r3-moto/moto-1b-script"
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16")
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True, language="ipt_Roma")

inputs = tok("La capitale de la France est", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

tok.set_language("tur_Latn")  # or: with tok.using("..."): ...

Use the tokenizer of the input's language, as in training. Without language, the tokenizer defaults to fw_edu. tok.languages lists the options. To send other languages (or tasks) to a tokenizer of your choice, use tok.set_mapping({"cat_Latn": "ipt_Roma", "regex:.*_Cyrl": "rus_Cyrl"}), then call tok.set_language("cat_Latn"). tok.mapping shows the full table.

Each language's tokenizer is shipped as the original, unmodified, in tokenizers/<language>/, together with super_mapping.json, which maps its token ids into the model's shared superset vocabulary of 165022 entries. tokenizer.py applies that mapping and the special tokens exactly as in training, so its ids index the model's embedding table directly. len(tok) is that size.

tok.decode accepts ids from any language's tokenizer, so generations that drift into another language's vocabulary decode fully. moto_config.json records the routing table, special ids and sources, and super_vocab.json holds the superset vocabulary.

Without remote code. Load the original and map its ids yourself:

import json
from huggingface_hub import hf_hub_download

lang = "ipt_Roma"
orig = AutoTokenizer.from_pretrained(repo, subfolder=f"tokenizers/{lang}")
to_model = {int(k): v for k, v in json.load(open(hf_hub_download(repo, f"tokenizers/{lang}/super_mapping.json"))).items()}
ids = [134449] + [to_model[i] for i in orig("La capitale de la France est", add_special_tokens=False)["input_ids"] if i in to_model]

Tokenizers

Language (training source)	Folder	Source
arb_Arab, fas_Arab	tokenizers/ipt_Arab	flexitok/bpe_script_Arab_16000
cmn_Hani, jpn_Jpan	tokenizers/flexitok--bpe_script_CmJp_16000	flexitok/bpe_script_CmJp_16000
ell_Grek	tokenizers/ell_Grek	flexitok/bpe_ltr_ell_Grek_8000_v2
fw_edu	tokenizers/fw_edu	flexitok/bpe_ltr_fw_edu_32000_v2
hun_Latn	tokenizers/hun_Latn	flexitok/bpe_ltr_hun_Latn_8000_v2
rus_Cyrl	tokenizers/rus_Cyrl	flexitok/bpe_ltr_rus_Cyrl_16000_v2
tur_Latn	tokenizers/tur_Latn	flexitok/bpe_ltr_tur_Latn_8000_v2
dan_Latn, deu_Latn, nld_Latn, swe_Latn	tokenizers/ipt_Germ	flexitok/bpe_script_Germ_32000
fra_Latn, ita_Latn, por_Latn, spa_Latn	tokenizers/ipt_Roma	flexitok/bpe_script_Roma_32000
ind_Latn, vie_Latn	tokenizers/flexitok--bpe_script_SEAS_16000	flexitok/bpe_script_SEAS_16000
ces_Latn, pol_Latn	tokenizers/ipt_Slav	flexitok/bpe_script_Slav_16000

fw_edu is English (FineWeb-Edu). language also accepts eng_Latn (→ fw_edu), en (→ fw_edu), cmn_Hans (→ cmn_Hani), zho_Hans (→ cmn_Hani), pes_Arab (→ fas_Arab).

Special tokens and sequence format

During training every document was encoded as BOS (id 134449) + text + EOS (id 104637). All tokenizers share these two ids.

• tok(text) prepends BOS, as in training.
• tok(text) does not append EOS. Append tok.eos_token_id yourself when scoring complete documents.
• There is no pad token. For batched inference set tok.pad_token = tok.eos_token (padding is on the left by default). The setting survives set_language.
• Decoding is lossless: clean_up_tokenization_spaces is false.

Training configuration

Trained with lingua. The full training config is in training/config.yaml, with cluster paths removed.

Also included:

• training/metrics.jsonl: per-step training metrics logged by lingua (loss, learning rate, throughput)

License

Released under MIT.

Citation

@inproceedings{altintas2026moto,
  title={{MoTo}: Mixture of Tokenizers Towards Fair Multilingual Language Modeling},
  author={G{\"u}l Sena Alt{\i}nta{\c{s}} and Colin Raffel},
  booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=avjeUhPuC6}
}