Instructions to use FrontiersMind/NandiBERT-150M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FrontiersMind/NandiBERT-150M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="FrontiersMind/NandiBERT-150M", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("FrontiersMind/NandiBERT-150M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
NandiBERT-150M
Hugging Face | GitHub | Launch Blog | Website | Technical Report (Coming Soon)
License: Apache-2.0 | Authors: FrontiersMind
NandiBERT is a family of open bidirectional encoders that reach state of the art in both English and Indic languages at their size, where earlier encoders were strong in one and weak in the other.| Model | Parameters | Hidden | Context | Description |
|---|---|---|---|---|
| NandiBERT-150M (this model) | 146M | 768 | 16K | Pre-trained encoder |
| NandiBERT-450M | 445M | 1,024 | 32K | Long-context pre-trained encoder |
For NandiBERT-150M, the parameter count is the encoder that you fine-tune (146M). The checkpoint also stores the untied MLM output layer (77M), so the model file holds 224M parameters.
Description
- One Model. From English to Indic and Beyond: NandiBERT-150M covers English, 13 Indic languages in native script (11 of them also in romanized form), with Southeast Asian languages and Middle-East Languages, so a single model replaces separate English and Indic encoders.
- Deep and efficient: 28 layers at 768 hidden size, with grouped-query attention, QK-norm and SwiGLU, sized for fast inference and low-cost fine-tuning.
- 16K context: A 16,384-token window with a 100K-token multilingual vocabulary shared across the NandiBERT family.
- Factorized embeddings. The 100K-token vocabulary is stored as a rank-192 factorized embedding, which cuts the embedding table from 77M to 19M parameters and puts that capacity into the encoder layers instead.
- Built to fine-tune: A strong starting point for classification, named-entity recognition, reranking and retrieval, loadable with Hugging Face
transformers.
Benchmark Results
Indic Benchmarks
| Model Name | Parameters | Classification | Structure Prediction | QA | Average | ||||
|---|---|---|---|---|---|---|---|---|---|
| IndicSentiment | IndicXNLI | IndicCOPA | IndicXParaphrase | Naamapadam NER (f1) | MASSIVE Slot-fill (f1) | IndicQA (f1) | |||
| IndicBERT-v2 | 272M | 91.2 | 73.7 | 64.6 | 56.9 | 73.2 | 58.1 | 48.3 | 66.57 |
| NandiBERT-150M | 146M | 91.3 | 75.9 | 66.4 | 60.6 | 77.4 | 54 | 46 | 67.37 |
Note: NandiBERT is pre-trained on 13 Indic languages in native script: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu.
English Benchmarks
| Model Name | Languages | Parameters | Classification | Retrieval | Average | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GLUE | SQuAD 2.0 | MUltiCoNERv2 | few-nerd | Reward Bench | BEIR | RAR-b Code | RAR-b Math | MLDR | LongEmbed | ||||
| BERT-base | English | 110M | 84.7 | 76.3 | 53.8 | 66.6 | 59.4 | 36.7 | 6.6 | 27.5 | 29 | 29.3 | 51.45 |
| RoBERTa-base | English | 125M | 86.4 | 83.7 | 56.5 | 67.1 | 63.2 | 39.6 | 11.7 | 33.4 | 28.7 | 33.8 | 55.20 |
| MosaicBERT-base | English | 137M | 85.4 | 79.4 | 57.8 | 65.7 | 62.5 | 41.4 | 12.7 | 30.9 | 29.6 | 40.5 | 54.48 |
| NomicBERT-2048 | English | 137M | 84 | 82.7 | 60.1 | 66.9 | 64.1 | 42 | 13.8 | 30.2 | 29.9 | 45 | 55.48 |
| GTE-en-MLM-base | English | 137M | 85.6 | 82.7 | 59.2 | 66.4 | 63.4 | 42.4 | 12 | 33.6 | 37.1 | 58.5 | 55.66 |
| ModernBERT-base | English | 149M | 88.4 | 86.7 | 57.4 | 66.7 | 68 | 42.6 | 29.6 | 46.1 | 37.7 | 63.2 | 60.69 |
| Ettin-Encoder-150M | English | 149M | 88.9 | 87.4 | 57.4 | 66.5 | 70.3 | 42.4 | 31.8 | 47.1 | 35.9 | 60.1 | 61.48 |
| UltraBERT-150M | English | 152M | 89.5 | 89.2 | 57.9 | 67.4 | 73 | 45.2 | 39.1 | 55.4 | 43.5 | 72 | 64.59 |
| NandiBERT-150M | Multi-Lingual | 146M | 87.1 | 87.6 | 56.9 | 68.1 | 71.2 | 44.5 | 32.8 | 49.4 | 41.2 | 71.1 | 62.20 |
Note: Model results are the average of 3 different seeds.
Usage
NandiBERT-150M loads with Hugging Face transformers 5.x using trust_remote_code=True.
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
model_id = "FrontiersMind/NandiBERT-150M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
trust_remote_code=True,
device_map="cuda",
)
texts = [
f"The capital of France is {tokenizer.mask_token}.",
f"भारत की राजधानी {tokenizer.mask_token} है।",
]
for text in texts:
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
logits = model(**inputs).logits
mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
print(tokenizer.decode(logits[0, mask_index].argmax()))
Fine-tuning
AutoModelForSequenceClassification and AutoModelForTokenClassification reuse the pretrained MLM head. The encoder output goes through the MLM head (dense, activation, RMSNorm) and then a new linear layer, so only the linear layer starts from random weights. This is the setup we recommend for classification and NER.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "FrontiersMind/NandiBERT-150M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=3, trust_remote_code=True)
For NER with the MLM head, use a learning rate of 3e-5. A low learning rate like 5e-6 gives much worse results. To fine-tune without the MLM head, pass classifier_head=None to from_pretrained.
For word-level input (is_split_into_words=True, as in NER), the tokenizer adds a space before every word, the same as in running text, so pass the words as they are. Use word_ids() to map tokens back to words.
Citation
@misc{frontiersmind2026nandibert,
title = {NandiBERT: Open Encoders for English and Indic Languages},
author = {{FrontiersMind}},
year = {2026},
howpublished = {\url{https://huggingface.co/FrontiersMind/NandiBERT-150M}}
}
- Downloads last month
- 173