Instructions to use FrontiersMind/NandiBERT-450M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FrontiersMind/NandiBERT-450M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="FrontiersMind/NandiBERT-450M", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("FrontiersMind/NandiBERT-450M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
NandiBERT-450M
Hugging Face | GitHub | Launch Blog | Website | Technical Report (Coming Soon)
License: Apache-2.0 | Authors: FrontiersMind
NandiBERT is a family of open bidirectional encoders that reach state of the art in both English and Indic languages at their size, where earlier encoders were strong in one and weak in the other.| Model | Parameters | Hidden | Context | Description |
|---|---|---|---|---|
| NandiBERT-150M | 146M | 768 | 16K | Pre-trained encoder |
| NandiBERT-450M (this model) | 445M | 1,024 | 32K | Long-context pre-trained encoder |
For NandiBERT-450M, the parameter count is the encoder that you fine-tune (445M). The checkpoint also stores the untied MLM output layer (103M), so the model file holds 548M parameters.
Description
- One Model. From English to Indic and Beyond: NandiBERT-450M covers English, 11 Indic languages in native script and romanized form, with Southeast Asian languages and Middle-East Languages, so a single model replaces separate English and Indic encoders.
- Deep, with efficient attention: 28 layers at 1,024 hidden size, with QK-norm and SwiGLU. Grouped-query attention with 8 query heads and 2 key-value heads keeps memory low on long inputs.
- 32K context: A 32,768-token window, so full contracts, reports and papers fit in a single pass, with a 100K-token multilingual vocabulary shared across the NandiBERT family.
- Factorized embeddings: The 100K-token vocabulary is stored as a rank-512 factorized embedding, which cuts the embedding table from 103M to 52M parameters and puts that capacity into the encoder layers instead.
- Built to fine-tune: A strong starting point for classification, named-entity recognition, reranking and long-document retrieval, loadable with Hugging Face
transformers.
Benchmark Results
Indic Benchmarks
| Model Name | Parameters | Classification | Structure Prediction | QA | Average | ||||
|---|---|---|---|---|---|---|---|---|---|
| IndicSentiment | IndicXNLI | IndicCOPA | IndicXParaphrase | Naamapadam NER (f1) | MASSIVE Slot-fill (f1) | IndicQA (f1) | |||
| IndicBERT-v2 | 272M | 88.3 | 74.3 | 63 | 57 | 72.4 | 57.3 | 49.2 | 65.93 |
| NandiBERT-450M | 445M | 92.9 | 81.3 | 76.1 | 77.8 | 72.5 | 61 | 51 | 73.23 |
Note: NandiBERT is pre-trained on 13 Indic languages in native script: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu.
English Benchmarks
| Model Name | Languages | Parameters | Classification | Retrieval | Average | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GLUE | SQuAD 2.0 | MUltiCoNERv2 | few-nerd | Reward Bench | BEIR | RAR-b Code | RAR-b Math | MLDR | LongEmbed | ||||
| BERT-large | English | 334M | 85.2 | 81.8 | 57.7 | 67.8 | 62.6 | 38.6 | 12.2 | 33 | 29.8 | 30 | 49.87 |
| RoBERTa-large | English | 355M | 88.9 | 89.4 | 62.3 | 68.1 | 68.5 | 42.8 | 20.3 | 37.8 | 29.5 | 32.2 | 53.98 |
| GTE-en-MLM-large | English | 434M | 87.6 | 87.2 | 63 | 67.1 | 62.2 | 39.4 | 13.5 | 32 | 40.8 | 57 | 54.98 |
| NeoBERT | English | 222M | 89 | 86.6 | 57.4 | 66.4 | 57.6 | 42.9 | 17.3 | 33.4 | 21.7 | 32.4 | 50.47 |
| LFM2.5-Encoder-350M | English | 355M | 89.2 | 87.2 | 62.7 | 67.9 | 68.5 | 47.3 | 38.6 | 50.2 | 44.5 | 71.7 | 62.78 |
| Ettin-Encoder-400M | English | 395M | 90.8 | 90.4 | 62.3 | 67.8 | 76.7 | 45.9 | 38.5 | 54 | 40.9 | 64.3 | 63.16 |
| ModernBERT-large | English | 395M | 90.4 | 89.8 | 60.9 | 67.6 | 72.8 | 45.2 | 34 | 46.1 | 39.9 | 63.5 | 61.02 |
| UltraBERT-380M | English | 380M | 91 | 90.9 | 63.5 | 68.3 | 78.8 | 47.7 | 47 | 61.7 | 44.1 | 74.9 | 66.79 |
| NandiBERT-450M | Multi-Lingual | 445M | 89.9 | 89.7 | 61.4 | 68.5 | 74.7 | 46.1 | 37.4 | 52.8 | 41.46 | 71.4 | 63.34 |
Note: Model results are the average of 3 different seeds.
Usage
NandiBERT-450M loads with Hugging Face transformers 5.x using trust_remote_code=True. For long documents, pass inputs of up to 32,768 tokens.
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
model_id = "FrontiersMind/NandiBERT-450M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
trust_remote_code=True,
device_map="cuda",
)
texts = [
f"The capital of France is {tokenizer.mask_token}.",
f"भारत की राजधानी {tokenizer.mask_token} है।",
]
for text in texts:
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
logits = model(**inputs).logits
mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
print(tokenizer.decode(logits[0, mask_index].argmax()))
Fine-tuning
AutoModelForSequenceClassification and AutoModelForTokenClassification reuse the pretrained MLM head. The encoder output goes through the MLM head (dense, activation, RMSNorm) and then a new linear layer, so only the linear layer starts from random weights. This is the default setup.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "FrontiersMind/NandiBERT-450M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=3, trust_remote_code=True)
To fine-tune without the MLM head, pass classifier_head=None to from_pretrained.
For word-level input (is_split_into_words=True, as in NER), the tokenizer adds a space before every word, the same as in running text, so pass the words as they are. Use word_ids() to map tokens back to words.
Citation
@misc{frontiersmind2026nandibert,
title = {NandiBERT: Open Encoders for English and Indic Languages},
author = {{FrontiersMind}},
year = {2026},
howpublished = {\url{https://huggingface.co/FrontiersMind/NandiBERT-450M}}
}
- Downloads last month
- 141