Instructions to use bammera-pothana/BERT-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use bammera-pothana/BERT-small with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="bammera-pothana/BERT-small")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForPreTraining tokenizer = AutoTokenizer.from_pretrained("bammera-pothana/BERT-small") model = AutoModelForPreTraining.from_pretrained("bammera-pothana/BERT-small", device_map="auto") - Notebooks
- Google Colab
- Kaggle
BERT-small
A BERT-Small model (4 layers, hidden size 512, 8 attention heads, 29.6 M parameters) pre-trained from scratch on Telugu with Google's original BERT recipe. It is the third member of the Google's Recipes collection, which reproduces the compact "BERT miniatures" of Turc et al. (2019) for Telugu, size by size, without changing the recipe. Its smaller siblings are BERT-tiny and BERT-mini.
It is intended as a compact but capable Telugu encoder: a strong starting point for fine-tuning where BERT-Base is too heavy, a point on a scaling curve with the other sizes in the collection, or a masked language model for scoring and ranking Telugu text.
Model details
| Architecture | BertForPreTraining (masked LM head + next-sentence head) |
| Size | BERT-Small: 4 layers, hidden 512, 8 heads, feed-forward 2048 |
| Parameters | 29,553,408 |
| Max sequence length | 512 |
| Vocabulary | 32,000 WordPiece tokens, cased, learned on the pre-training text; identical to the BERT-mini vocabulary |
| Precision of the released weights | float32 (model.safetensors, 119 MB) |
| Language | Telugu (te) |
| Script | Telugu script, NFC-normalised, no lower-casing or accent stripping |
| Trained by | bammera-pothana |
| Training dates | 10 – 12 November 2026 |
The tokenizer is a standard BertTokenizerFast. Lower-casing, accent stripping and CJK handling are off: stripping accents would destroy Telugu vowel signs (మాత్రలు), so the tokenizer must be loaded from this repository rather than configured by hand. BERT-small and BERT-mini share the same vocabulary and the same pre-training text, so their losses compare directly; BERT-tiny has its own vocabulary, so its losses are only roughly comparable.
Training recipe
The recipe follows Devlin et al. (2018), as used for the BERT miniatures of Turc et al. (2019), with no modifications other than the Telugu vocabulary:
| Setting | Value |
|---|---|
| Objectives | Masked LM (15 % of WordPiece tokens: 80 % [MASK], 10 % random token, 10 % unchanged) + next-sentence prediction |
| Masking | Token-level (Google's original BERT), dynamic, re-sampled every batch |
| Total steps | 1,000,000 at a global batch of 256 sequences, one continuous schedule |
| Phase 1 | 900,000 steps at 128 tokens |
| Phase 2 | 100,000 steps at 512 tokens, optimizer state carried over from phase 1 |
| Optimizer | AdamW, lr 1e-4, β₁ 0.9, β₂ 0.999, ε 1e-6, weight decay 0.01 (not on bias / LayerNorm) |
| Schedule | Linear warm-up over the first 10,000 steps, then linear decay to 0 at step 1,000,000 |
| Gradient clipping | 1.0 |
| Dropout | 0.1 (hidden and attention) |
| Activation | GELU |
| Early stopping | Off; the checkpoint with the best held-out MLM loss is released |
| Mixed precision | float16 autocast with loss scaling; weights kept in float32 |
| Hardware | 4 × NVIDIA GeForce RTX 2080 Ti (11 GB) |
| Wall-clock | 27.0 h for phase 1 (0.11 s/step) + 12.4 h for phase 2 (0.44 s/step) |
Training examples were built the way Google's create_pretraining_data.py builds them: [CLS] A [SEP] B [SEP], where B is the true continuation half of the time and a segment from a random other document otherwise; 10 % of examples use a shorter random target length. A held-out set of documents was kept aside before the vocabulary was learned and was never used for training.
Evaluation
Held-out pre-training text
Held-out documents, masked with a fixed random seed so that numbers are comparable across checkpoints. MLM accuracy is top-1 accuracy at masked positions.
| Checkpoint | MLM loss | MLM perplexity | MLM accuracy | NSP accuracy |
|---|---|---|---|---|
| End of phase 1 (step 900 k, 128 tokens) | 2.933 | 18.8 | 47.8 % | 95.4 % |
| Released: end of phase 2 (step 1 M, 512 tokens) | 2.830 | 16.9 | 49.4 % | 96.3 % |
| BERT-mini, released checkpoint (same vocabulary and text) | 3.418 | 30.5 | 42.4 % | 95.5 % |
Held-out MLM loss had converged by the end of phase 2 (2.837 at step 60 k → 2.830 at step 100 k). The per-phase evaluation files are in the training/ folder of this repository.
Clean typed Telugu outside the pre-training text
Two fixed samples of 3,000 paragraphs each, cut into 128-token pieces, 15 % of tokens masked with a fixed seed. Neither set was used in pre-training. The other sizes in the collection are shown for scale.
| Model | Wikisource (literary): loss / accuracy | News (modern): loss / accuracy |
|---|---|---|
| BERT-small (this model) | 3.05 / 46.2 % | 2.78 / 50.7 % |
| BERT-mini (4 layers, 256 hidden, same vocabulary) | 3.61 / 39.4 % | 3.33 / 43.8 % |
| BERT-tiny (2 layers, 128 hidden, own vocabulary) | 4.55 / 29.7 % | 4.73 / 28.7 % |
Each step up the collection has added roughly 7 points of masked-token accuracy on both sets, with the news set, the one furthest from the pre-training text in register, gaining slightly more.
Downstream (fine-tuning) results for this model will be added to this card as they become available.
How to use
from transformers import pipeline
fill = pipeline("fill-mask", model="bammera-pothana/BERT-small")
fill("ఆంధ్ర [MASK] పత్రిక")
# [{'token_str': 'వార', ...}, {'token_str': '##జ్యోతి', ...}, {'token_str': 'సచిత్ర', ...}, ...]
As an encoder for fine-tuning:
from transformers import AutoTokenizer, AutoModel, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("bammera-pothana/BERT-small")
enc = AutoModel.from_pretrained("bammera-pothana/BERT-small") # hidden states, pooler
clf = AutoModelForSequenceClassification.from_pretrained("bammera-pothana/BERT-small", num_labels=3)
Loading with AutoModelForMaskedLM discards the pooler and next-sentence head, which is expected and harmless. The full pre-training heads are available through BertForPreTraining.
Sample predictions from the released checkpoint:
| Input | Top predictions |
|---|---|
నేను [MASK] వెళ్తున్నాను. |
ఢిల్లీ (0.05), హైదరాబాదు, విశాఖపట్నం, గుంటూరు, మద్రాసు |
ఆంధ్ర [MASK] పత్రిక |
వార (0.26, as in ఆంధ్ర వారపత్రిక), ##జ్యోతి (as in ఆంధ్రజ్యోతి), సచిత్ర, రైతు, పత్రిక |
తెలుగు [MASK] భాష |
భాష (0.04), ##వారి, ##భాష, ., - |
The third prompt is a hard one for every size in the collection: the model spreads its probability over several completions and punctuation, which is a reminder that these are token probabilities on short contexts, not knowledge.
Limitations and bias
- Compact model. With 4 layers and hidden size 512, BERT-Small is the strongest of the three compact sizes released so far but still well below BERT-Base on downstream tasks. Use it where speed or size matters, or as a point on a scaling curve.
- Domain. Like any pre-trained language model, it reflects the register, spelling conventions and viewpoints of the text it was trained on, and may perform worse on text from other domains. The sample predictions above show that it has learned older spellings such as మద్రాసు alongside modern ones.
- Pre-training only. No instruction tuning, no safety tuning. Outputs are token probabilities, not answers.
Repository contents
| File | Purpose |
|---|---|
config.json, model.safetensors |
Model (architecture BertForPreTraining) |
tokenizer.json, tokenizer_config.json, special_tokens_map.json, vocab.txt |
WordPiece tokenizer |
training/eval_phase1_128.json, training/eval_phase2_512.json |
Held-out metrics at the end of each phase |
Citation
If you use this model, please cite the original BERT and BERT-miniatures papers together with this repository:
@misc{bammera-pothana-bert-small-2026,
title = {BERT-small: a Telugu BERT-Small model pre-trained with Google's original recipe},
author = {bammera-pothana},
year = {2026},
url = {https://huggingface.co/bammera-pothana/BERT-small}
}
@article{turc2019,
title = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
journal = {arXiv preprint arXiv:1908.08962},
year = {2019}
}
@inproceedings{devlin2019,
title = {{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding},
author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
booktitle = {NAACL-HLT},
year = {2019}
}
- Downloads last month
- 39
Collection including bammera-pothana/BERT-small
Paper for bammera-pothana/BERT-small
Evaluation results
- Masked-LM loss (cross-entropy) on Held-out Telugu text (512-token examples)self-reported2.830
- Masked-LM perplexity on Held-out Telugu text (512-token examples)self-reported16.900
- Masked-LM top-1 accuracy on Held-out Telugu text (512-token examples)self-reported0.494
- Next-sentence-prediction accuracy on Held-out Telugu text (512-token examples)self-reported0.963
- Masked-LM loss (cross-entropy) on Telugu Wikisource (typed literary text, 3,000 paragraphs, 128-token pieces)self-reported3.050
- Masked-LM top-1 accuracy on Telugu Wikisource (typed literary text, 3,000 paragraphs, 128-token pieces)self-reported0.462
- Masked-LM loss (cross-entropy) on Telugu news (typed modern text, 3,000 paragraphs, 128-token pieces)self-reported2.778
- Masked-LM top-1 accuracy on Telugu news (typed modern text, 3,000 paragraphs, 128-token pieces)self-reported0.507