NandiBERT-450M

NandiBERT-450M by FrontiersMind

Hugging Face | GitHub | Launch Blog | Website | Technical Report (Coming Soon)

License: Apache-2.0 | Authors: FrontiersMind

NandiBERT is a family of open bidirectional encoders that reach state of the art in both English and Indic languages at their size, where earlier encoders were strong in one and weak in the other.
Model Parameters Hidden Context Description
NandiBERT-150M 146M 768 16K Pre-trained encoder
NandiBERT-450M (this model) 445M 1,024 32K Long-context pre-trained encoder

For NandiBERT-450M, the parameter count is the encoder that you fine-tune (445M). The checkpoint also stores the untied MLM output layer (103M), so the model file holds 548M parameters.

Description

  • One Model. From English to Indic and Beyond: NandiBERT-450M covers English, 11 Indic languages in native script and romanized form, with Southeast Asian languages and Middle-East Languages, so a single model replaces separate English and Indic encoders.
  • Deep, with efficient attention: 28 layers at 1,024 hidden size, with QK-norm and SwiGLU. Grouped-query attention with 8 query heads and 2 key-value heads keeps memory low on long inputs.
  • 32K context: A 32,768-token window, so full contracts, reports and papers fit in a single pass, with a 100K-token multilingual vocabulary shared across the NandiBERT family.
  • Factorized embeddings: The 100K-token vocabulary is stored as a rank-512 factorized embedding, which cuts the embedding table from 103M to 52M parameters and puts that capacity into the encoder layers instead.
  • Built to fine-tune: A strong starting point for classification, named-entity recognition, reranking and long-document retrieval, loadable with Hugging Face transformers.

Benchmark Results

Indic Benchmarks

Model NameParametersClassificationStructure PredictionQAAverage
IndicSentimentIndicXNLIIndicCOPAIndicXParaphraseNaamapadam NER (f1)MASSIVE Slot-fill (f1)IndicQA (f1)
IndicBERT-v2272M88.374.3635772.457.349.265.93
NandiBERT-450M445M92.981.376.177.872.5615173.23

Note: NandiBERT is pre-trained on 13 Indic languages in native script: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu.

English Benchmarks

Model NameLanguagesParametersClassificationRetrievalAverage
GLUESQuAD 2.0MUltiCoNERv2few-nerdReward BenchBEIRRAR-b CodeRAR-b MathMLDRLongEmbed
BERT-largeEnglish334M85.281.857.767.862.638.612.23329.83049.87
RoBERTa-largeEnglish355M88.989.462.368.168.542.820.337.829.532.253.98
GTE-en-MLM-largeEnglish434M87.687.26367.162.239.413.53240.85754.98
NeoBERTEnglish222M8986.657.466.457.642.917.333.421.732.450.47
LFM2.5-Encoder-350MEnglish355M89.287.262.767.968.547.338.650.244.571.762.78
Ettin-Encoder-400MEnglish395M90.890.462.367.876.745.938.55440.964.363.16
ModernBERT-largeEnglish395M90.489.860.967.672.845.23446.139.963.561.02
UltraBERT-380MEnglish380M9190.963.568.378.847.74761.744.174.966.79
NandiBERT-450MMulti-Lingual445M89.989.761.468.574.746.137.452.841.4671.463.34

Note: Model results are the average of 3 different seeds.

Usage

NandiBERT-450M loads with Hugging Face transformers 5.x using trust_remote_code=True. For long documents, pass inputs of up to 32,768 tokens.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
 
model_id = "FrontiersMind/NandiBERT-450M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="cuda",
)
 
texts = [
    f"The capital of France is {tokenizer.mask_token}.",
    f"भारत की राजधानी {tokenizer.mask_token} है।",
]
 
for text in texts:
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**inputs).logits
    mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
    print(tokenizer.decode(logits[0, mask_index].argmax()))

Fine-tuning

AutoModelForSequenceClassification and AutoModelForTokenClassification reuse the pretrained MLM head. The encoder output goes through the MLM head (dense, activation, RMSNorm) and then a new linear layer, so only the linear layer starts from random weights. This is the default setup.

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "FrontiersMind/NandiBERT-450M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=3, trust_remote_code=True)

To fine-tune without the MLM head, pass classifier_head=None to from_pretrained.

For word-level input (is_split_into_words=True, as in NER), the tokenizer adds a space before every word, the same as in running text, so pass the words as they are. Use word_ids() to map tokens back to words.

Citation

@misc{frontiersmind2026nandibert,
  title        = {NandiBERT: Open Encoders for English and Indic Languages},
  author       = {{FrontiersMind}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/FrontiersMind/NandiBERT-450M}}
}
Downloads last month
141
Safetensors
Model size
0.5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including FrontiersMind/NandiBERT-450M