UltraBERT: One Encoder for Classification and Retrieval

TL;DR

We're releasing UltraBERT, a family of English encoders (150M and 380M) with a native 32K context, plus UltraBERT-Embed-150M, a ready-to-use embedding model built on top of it. Everything is Apache 2.0 and lives in the UltraBERT collection.

• Encoders stop improving after 1–2T tokens, even while the loss keeps going down. More compute alone won't save them.
• The masking ratio decides what an encoder is good at. Low masking wins classification, high masking wins retrieval. Every existing encoder picks one.
• UltraBERT is built for both. We mix 5% and 50% masking in every batch, give each ratio its own LM head, and distill from a 1.3B teacher only where it helps.
• Top score on 9 of 10 benchmarks at both sizes, including 91.0 on GLUE and 47.7 on BEIR for the 380M model, with 99% passkey accuracy all the way to 32K tokens.
• UltraBERT-Embed-150M scores 67.24 on MTEB(eng, v2), the best dense embedding model under 200M parameters, trained without distillation from a larger embedder.

You can try it right away:

pip install -U "transformers>=5" "sentence-transformers>=6.0"

from transformers import pipeline

pipe = pipeline("fill-mask", model="dmis-lab/UltraBERT-380M", trust_remote_code=True)
pipe("The capital of France is <|mask|>.")

Where Encoders Stand Today

Encoders are still the most cost-effective way to process text at volume. Most RAG pipelines embed their documents with one, and pretraining datasets like FineWeb-Edu were filtered by having one score their quality. An encoder reads the input once, in both directions, and returns a vector or a score. There is no token-by-token generation, which is why it is so cheap to run.

Of course, LLMs are remarkable generalists. Given a prompt and a few examples, they handle a wide range of tasks reasonably well without any fine-tuning. However, when the task is well defined and there is enough labeled data to learn it, fine-tuning a much smaller encoder usually works better (Bucher & Martini, 2024; Edwards & Camacho-Collados, 2024; Zhang et al., 2025). For non-generative tasks, encoders deliver far better performance for their size (Weller et al., 2026).

More recently, decision models such as Jev have brought renewed attention to this kind of work: reading an input once and choosing among a fixed set of answers without generating text. That is exactly what encoders are good at.

All of these uses lean on a strong pretrained backbone. Over the last two years, encoder backbones finally caught up architecturally. The training recipe, however, still follows conventions from the BERT era. The masked language modeling objective is configured with lessons from studies that trained on 10B–100B tokens. On current leaderboards, the encoder that ranks first on GLUE is not the one that ranks first on BEIR, and no single model leads both.

We wanted to know whether this trade-off is a law of nature or an artifact of how encoders are trained. So we went back and re-examined the recipe at a scale nobody had tried.

Three Lessons From Training Encoders At Scale

We trained encoders at 150M, 380M and 1.3B parameters with a modern architecture (RoPE, SwiGLU, alternating local/global attention) on the same data mixture, and fine-tuned and evaluated every checkpoint the same way. Counting every run, these ablations add up to more than 70 trillion training tokens, with the longest single run going to 8.8T. Three findings came out of it.

1. Decoders Scale. Encoders Saturate.

Decoders are famous for improving almost indefinitely with more tokens. Encoders, it turns out, are not. GLUE and SQuAD plateau around 1–2T tokens, and BEIR plateaus even earlier, while both training and validation loss keep dropping all the way to 8.8T.

We double-checked this against a decoder of similar size (SmolLM2-1.7B) on 6 standard MCQA benchmarks. After instruction tuning, our encoder flattens out just like on the other tasks, while the decoder keeps climbing. Past saturation, training an encoder on more tokens with the same recipe brings little benefit. Further gains require changing the recipe itself.

2. Classification and Retrieval Favor Opposite Masking Ratios

Prior work concluded that larger models want higher masking ratios. But those studies ran on budgets where a higher ratio mainly meant faster convergence. Once every model is trained to saturation, that confound disappears, and we see the opposite: classification gets steadily worse as you mask more, at every model size.

Retrieval goes the other way. 50% masking gives the best BEIR score at every scale, and the representations use more of their dimensions (higher effective rank) as the ratio goes up. We tried to get that benefit cheaply with the KoLeo regularizer at 6% masking. It raises BEIR a bit, falls short of plain 50% masking, and costs GLUE points. No single ratio serves both.

Going Deeper: What ELECTRA-Style Training Reveals About Masking

ELECTRA-style models, trained with Replaced Token Detection (RTD), are known to beat MLM on classification, and they still do against modern MLM encoders, though at a heavy cost on retrieval. The usual explanation is training efficiency. Training to saturation removes efficiency as a factor, which leaves one main difference: how the input is corrupted. RTD's generator replaces tokens with plausible alternatives that fit the context, and these carry more information than a bare mask token. We suspected this contextual corruption is what really gives RTD its edge, so we compared MLM and RTD on 150M encoders at matched corruption levels.

The gap shrinks at the two ends for different reasons. At 50%, the generator sees too little context, so its replacements become trivial. At 3%, so few tokens are corrupted that it hardly matters whether they are plausible words or mask tokens. Since classification favors low masking anyway, you can stay with MLM there, keep the classification quality, and avoid the retrieval penalty that RTD encoders are known for.

3. Distillation Helps Only at Low Masking Ratios

Pruning a larger model and distilling it into a smaller one is a common way to build strong small models, for encoders and decoders alike. We tested whether a 1.3B teacher helps equally at both masking ratios. At 15% masking, distillation lifted GLUE (90.2 → 90.8 at 380M). At 50% masking, however, it hurt retrieval: the distilled model lost BEIR points compared with training from scratch (47.2 → 46.5). With half of the input masked, the teacher's predictions carry little signal beyond what plain MLM already provides. So the rule became simple: distill at the low ratio, train the high ratio with plain MLM.

The UltraBERT Recipe

If classification wants 5% masking and retrieval wants 50%, the obvious move is to train on both. We tried the obvious variants first: decaying the ratio from high to low, and sampling it uniformly per batch. Neither works. Uniform sampling just behaves like its average (about 30%), and decay never gets back the classification quality of a low ratio (88.8 vs. 89.5 GLUE). Mixing 5% and 50% sequences in the same batch got closer, so we looked at what was happening inside the model.

The two regimes want the output layer to look completely different, and since every gradient the encoder receives flows back through that head, sharing it forces them to fight. We resolve this conflict with LM head decoupling:

LM head decoupling. Each batch mixes 5% and 50% masked sequences at a 4:1 ratio, and each ratio gets its own LM head. The encoder body is shared. The heads are not.

With decoupled heads, one 150M encoder reaches 89.4 GLUE and 45.1 BEIR, matching the best single-ratio model on classification (89.5 at 6%) and beating the best one on retrieval (45.0 at 50%).

Strategy (150M)	GLUE	BEIR
Single ratio, 6%	89.5	43.5
Single ratio, 50%	87.5	45.0
Uniform, 5%–50%	88.0	44.2
Mix 5% + 50%, shared head	88.9	44.7
Mix 5% + 50%, decoupled heads	89.4	45.1

Put together with the distillation rule, the full pipeline looks like this. Following Cascade Distillation (Muralidharan et al., 2024; Liu et al., 2026), we start from our best 1.3B encoder, width-prune it to 380M and distill it, then prune the distilled 380M model to 150M and distill again. Both students learn directly from the 1.3B teacher rather than from the intermediate 380M model. The 5% head learns from the teacher's logits with a KL loss; the 50% head trains with standard MLM. Training runs on 2T tokens, the last 100B of them at 32K context. The pretraining mix is mostly web text, plus code, academic papers, math and a small slice of instruction data. That 2.5% of instruction data matters more than its size suggests: the base model already follows MCQA prompts zero-shot through a single mask token.

Performance

We compared UltraBERT against recent encoder families, fine-tuning each one per task with the same grid search. It takes the top score on 9 of 10 benchmarks at both sizes. The biggest jumps are on reasoning-style retrieval: +7.3 and +8.3 points on RAR-b Code and Math at base size, and +8.4 and +7.7 at medium.

More importantly, the same model now leads on both classification and retrieval.

Base (≤200M)

Medium (≤500M)

Model	Params	GLUE	SQuAD 2.0	MultiCoNER	Few-NERD	RewardBench	BEIR	RAR-b Code	RAR-b Math	MLDR	LongEmbed
BERT	334M	85.2	81.8	57.7	67.8	62.6	38.6	12.2	33.0	29.8	30.0
RoBERTa	355M	88.9	89.4	62.3	68.1	68.5	42.8	20.3	37.8	29.5	32.2
GTE-en-MLM	434M	87.6	87.2	63.0	67.1	62.2	39.4	13.5	32.0	40.8	57.0
NeoBERT	222M	89.0	86.6	57.4	66.4	57.6	42.9	17.3	33.4	21.7	32.4
LFM2.5-Encoder	230M	88.2	86.1	59.6	67.6	67.5	46.7	30.9	50.2	43.7	70.0
LFM2.5-Encoder	355M	89.2	87.2	62.7	67.9	68.5	47.3	38.6	50.2	44.5	71.7
ModernBERT	395M	90.4	89.8	60.9	67.6	72.8	45.2	34.0	46.1	39.9	63.5
Ettin Encoder	395M	90.8	90.4	62.3	67.8	76.7	45.9	38.5	54.0	40.9	64.3
UltraBERT	380M	91.0	90.9	63.5	68.3	78.8	47.7	47.0	61.7	44.1	74.9

A couple of comparisons we're especially happy with: UltraBERT-150M matches RoBERTa-large on classification with 2.3× fewer parameters, and beats Ettin-400M on retrieval at 2.6× smaller size.

Reliable Retrieval Across the Full 32K Context

The passkey test on LongEmbed measures how much of a model's context window is actually usable. Models with a 512-token context fail beyond 1K tokens, 8K-context models drop sharply at 16K and 32K, and LFM2.5-Encoder degrades more gently but falls below 90% at 32K. UltraBERT holds 99% at every length up to 32K, at both sizes.

Long inputs matter beyond retrieval, too. Classification and decision workloads often need to judge a whole document or conversation at once.

UltraBERT-Embed-150M

The backbone numbers above show what UltraBERT can do with a light fine-tune. To show what happens with a full embedding pipeline, we're also releasing UltraBERT-Embed-150M. It is trained from UltraBERT-150M in two contrastive stages: about 500M weakly supervised text and code pairs, then 5.7M curated examples with mined hard negatives and inputs up to 32K tokens. Public data only, and no distillation from a larger embedding model.

On MTEB(eng, v2) it scores 67.24, the highest of any dense model under 200M parameters. It also leads on code (MTEB Code 73.30), long documents (LongEmbed 79.30), and reasoning-style retrieval (RAR-b 28.75). On LongEmbed it embeds 32K-token documents whole instead of truncating them.

Model	Params	MTEB(eng, v2)	MTEB(Code, v1)	LongEmbed	RAR-b
UltraBERT-Embed-150M	151M	67.24	73.30	79.30	28.75
Tarka-Embedding-150M-V1 (distilled)	156M	66.39	57.71	55.12	21.76
gte-modernbert-base	149M	66.26	71.66	57.0	23.54
F2LLM-v2-160M (distilled)	159M	65.93	70.38	57.64	19.62
bge-base-en-v1.5	109M	65.14	50.35	33.9	14.59
granite-embedding-english-r2	149M	62.84	57.22	67.8	20.65
e5-base-v2	109M	62.67	53.51	41.1	23.83
DenseOn	149M	62.47	55.96	66.50	24.12
nomic-embed-text-v1.5	137M	62.20	47.36	30.60	15.39
modernbert-embed-base	149M	62.1	53.28	56.3	16.04

How to Use

All models need transformers>=5 and trust_remote_code=True. FlashAttention 2 is recommended for long inputs; SDPA works too. The released checkpoints contain the encoder and the 5% (distilled) MLM head, with task classes for sequence classification, token classification and question answering.

Masked Language Modeling

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "dmis-lab/UltraBERT-380M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # or "sdpa"
    device_map="cuda",
)

text = f"The capital of France is {tokenizer.mask_token}."
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    logits = model(**inputs).logits

mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
print(tokenizer.decode(logits[0, mask_index].argmax()))

Fine-Tuning

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "dmis-lab/UltraBERT-150M", num_labels=2, trust_remote_code=True
)

Tip: UltraBERT likes lower learning rates than other encoders, by up to 10×. For retrieval fine-tuning our best values were 5e-5 (150M) and 3e-5 (380M), where most encoders prefer 3e-4 to 5e-4.

Embeddings With Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "dmis-lab/UltraBERT-Embed-150M",
    trust_remote_code=True,
    model_kwargs={"attn_implementation": "flash_attention_2", "dtype": "bfloat16"},
)

queries = ["Win big, click now!"]
labels = ["spam", "promotional", "announcement"]
query_embeddings = model.encode(queries, prompt="Instruct: Labeling spam emails\nQuery: ")
label_embeddings = model.encode(labels, prompt="Label: ")
print(model.similarity(query_embeddings, label_embeddings))

Queries take an Instruct: {task}\nQuery: prefix and documents are embedded as-is. prompt_name="query" applies the default retrieval instruction.

Conclusion

The main lesson for us is that the "specialist encoder" was never inherent to encoders. It was a side effect of a recipe inherited from the BERT era. Once you train past saturation and let each masking regime keep its own head, one model can be good at both. We're excited to see what the community builds on top: guardrails that screen LLM traffic at scale, code and math search, and decision models that judge a full 32K-token input in a single pass.

If you try UltraBERT, we'd love to hear how it does on your task. Open a discussion on any of the model pages.

Links

• Models
  • UltraBERT-150M: pre-trained encoder, 32K context
  • UltraBERT-380M: pre-trained encoder, 32K context
  • UltraBERT-Embed-150M: ready-to-use embedding model
• Collection: dmis-lab/ultrabert
• Paper: UltraBERT: Beyond the Limit of BERT Pre-Training

Citation

@misc{park2026ultrabert,
      title={UltraBERT: Beyond the Limit of BERT Pre-Training}, 
      author={Jungwoo Park and Taewhoo Lee and Jaewoo Kang},
      year={2026},
      eprint={2610.15028},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.15028}, 
}