---
language:
- bn
- en
license: gemma
library_name: transformers
pipeline_tag: text-generation
base_model: google/gemma-3-1b-it
base_model_relation: finetune
tags:
- bangla
- bengali
- gemma3
- unsloth
- full-finetune
- instruction-tuned
datasets:
- md-nishat-008/Bangla-Instruct
metrics:
- accuracy
- perplexity
model-index:
- name: BanglaGPT-Pro
results:
- task:
type: text-generation
name: Bangla multiple-choice understanding
dataset:
name: hishab Bangla benchmark suite (bangla-mmlu, commonsenseqa-bn, openbookqa-bn, piqa-bn, boolq-bn)
type: hishab/bangla-mmlu
metrics:
- type: accuracy
name: Mean accuracy across five benchmarks
value: 0.380
- type: perplexity
name: Perplexity on held-out Bangla-Instruct responses
value: 14.51
---
[Hugging Face](https://huggingface.co/emonHasan01) | [GitHub](https://github.com/Md-Emon-Hasan) | [LinkedIn](https://www.linkedin.com/in/md-emon-hasan-695483237/) | [WhatsApp](https://wa.me/8801834363533) | [Portfolio](https://emonlabs-ai.hitechparks.com/)
**Author:** [Md. Emon Hasan](https://huggingface.co/emonHasan01)
# BanglaGPT-Pro
**A full fine-tune of [`google/gemma-3-1b-it`](https://huggingface.co/google/gemma-3-1b-it) for Bangla.**
Every parameter was trained — not LoRA, not an adapter — on
[`Bangla-Instruct`](https://huggingface.co/datasets/md-nishat-008/Bangla-Instruct),
80,000 instruction-response pairs written natively in Bangla rather than translated from
English.
**Perplexity on held-out Bangla falls from 26.10 to 14.51, a 44% reduction.** That is the headline because it is the only measure here without a chance floor. The multiple-choice benchmarks below sit near theirs for both models and separate almost nothing — at 1,000 items the standard error of a difference is ±0.022, and every per-benchmark change is inside that.
## Results
Five Bangla multiple-choice benchmarks at **1,000 items each**, scored by
**length-normalised log-likelihood** — the mean log probability of each option's tokens
given the prompt, highest wins. No generation, no LLM judge, no string matching.
| Benchmark | Chance | `gemma-3-1b-it` | **this model** | Δ | `llama-3.2-1b` † | `titulm-1b-v2.0` † |
|---|---|---|---|---|---|---|
| bangla-mmlu | 0.25 | 0.287 | **0.285** | -0.002 *(noise)* | 0.29 | 0.25 |
| commonsenseqa-bn | 0.20 | 0.278 | **0.264** | -0.014 *(noise)* | 0.22 | 0.26 |
| openbookqa-bn | 0.25 | 0.290 | **0.295** | +0.005 *(noise)* | 0.33 | 0.32 |
| piqa-bn | 0.50 | 0.535 | **0.541** | +0.006 *(noise)* | 0.53 | 0.58 |
| boolq-bn | 0.50 | 0.535 | **0.517** | -0.018 *(noise)* | 0.55 | — |
| **average** | — | 0.385 | **0.380** | -0.005 | — | — |
The first four columns were measured by this notebook. † marks figures **published by
Hishab** on the [`titulm-llama-3.2-1b-v2.0`](https://huggingface.co/hishab/titulm-llama-3.2-1b-v2.0)
card (0-shot), reproduced here, not re-measured.
**Same datasets, different measurement.** Those two columns are on exactly these five
datasets — TituLLM built them — which is why they are in this table at all. But they were
produced with `lm-evaluation-harness`, which feeds the raw question with no chat template
and normalises by character length, while the columns here use the model's chat template
with a Bangla system prompt and normalise by token count. Both are standard; they are not
the same measurement, and the gap between two such setups is commonly a few points. Read
the † columns for **scale**, not for ranking. A dash means Hishab did not report that cell.
What those columns do show cleanly is a pattern rather than a score: their Bangla model
lands **below** the base Llama on Bangla MMLU (0.25 vs 0.29) after 37 billion tokens of
continual pretraining. Whatever the Δ column here turns out to be, that shape is not
peculiar to this model.
**Read the chance column before the accuracy column.** At 1B scale these benchmarks sit
close to guessing for every model in the table. The standard error of a difference between
two columns at 1,000 items is ±0.022, so changes smaller than about 0.04 carry
no information; those are marked *(noise)*.
### Benchmark provenance
| Benchmark | Items used | Splits drawn from | Chance |
|---|---|---|---|
| bangla-mmlu | 1,000 | test(14,750) | 0.25 |
| commonsenseqa-bn | 1,000 | validation(1,221) | 0.20 |
| openbookqa-bn | 1,000 | test(497) + validation(500) + train(4,947) | 0.25 |
| piqa-bn | 1,000 | validation(1,838) | 0.50 |
| boolq-bn | 1,000 | test(729) + validation(432) | 0.50 |
Three of these sets have fewer than 1,000 rows in their test split, so splits were
concatenated to reach the target; the table says which. Three of the five
(`commonsenseqa-bn`, `openbookqa-bn`, `piqa-bn`) are **machine translations of English
datasets** refined by an LLM, per Hishab's cards — worth knowing when reading any Bangla
score built on them.
### Perplexity
| | perplexity |
|---|---|
| `gemma-3-1b-it` | 26.10 |
| **BanglaGPT-Pro** | **14.51** |
Measured on 200 held-out `Bangla-Instruct` responses (shuffled rows
200,000–200,800), far past the slice training drew from. Unlike the
multiple-choice tables this has no floor to sit near, which is why it is the headline.
### Appendix: figures published elsewhere, on different datasets
Read the caveats before the table, because the table is the part that invites a mistake.
**These numbers were not measured here and share no dataset with anything above.** They
come from Table 2 of the [TigerLLM paper](https://aclanthology.org/2025.acl-short.69/) as
reproduced on the [`TigerLLM-1B-it`](https://huggingface.co/md-nishat-008/TigerLLM-1B-it)
card, which evaluates on MMLU-bn (a Bangla slice of MMLU-Pro), PangBench-bn, BanglaQuaD,
mHumanEval-bn, BEnQA and BanglaRQA. Not one of those is one of the five used above. No row
for this model appears below, and none should: a row placed in that table would be the most
misleading thing on this page.
**Three specific problems with it**, beyond the dataset mismatch:
1. It scores Titu-LLM at 0.06 on MMLU-bn, where Hishab's own card reports 0.25 on Bangla
MMLU. 0.06 also sits below the 0.25 chance floor of a four-option benchmark — a figure a
working log-likelihood harness cannot really produce.
2. It places a 1B model at 0.61 against GPT-4o-mini's 0.67 on the same benchmark.
3. The current Gemma-3 TigerLLM checkpoints carry no numbers in it at all, only a "+2–4%"
claim relative to a deprecated LLaMA version.
The ACL version of that paper also notes its results are not reproducible.
So why reproduce it? Because "how does this compare to other Bangla models?" is a fair
question, and the honest answer is that two incompatible benchmark ecosystems exist for
Bangla and most published figures are not comparable with each other. Showing that is more
useful than silence. It is kept here, below the results, labelled, and with the problems
stated first.
| Model | MMLU-bn | PangBench-bn | BanglaQuaD | mHumanEval-bn | BEnQA | BanglaRQA |
|---|---|---|---|---|---|---|
| GPT-3.5 | 0.55 | 0.55 | 0.50 | 0.56 | 0.50 | 0.49 |
| Gemini 1.5 Flash | 0.66 | 0.57 | 0.62 | 0.58 | 0.56 | 0.61 |
| GPT-4o-mini | 0.67 | 0.62 | 0.65 | 0.56 | 0.60 | 0.60 |
| LLaMA 3.2 (11B) | 0.22 | 0.19 | 0.21 | 0.15 | 0.18 | 0.20 |
| Gemma 2 (27B) | 0.35 | 0.51 | 0.43 | 0.64 | 0.50 | 0.56 |
| Pangea (7B) | 0.18 | 0.15 | 0.17 | 0.10 | 0.14 | 0.16 |
| Titu-LLM | 0.06 | 0.19 | 0.08 | 0.02 | 0.17 | 0.21 |
| Bong-LLaMA | 0.05 | 0.12 | 0.08 | 0.02 | 0.15 | 0.13 |
| Bangla-LLaMA | 0.02 | 0.08 | 0.05 | 0.10 | 0.11 | 0.09 |
| Bangla-Gemma | 0.18 | 0.15 | 0.12 | 0.10 | 0.22 | 0.19 |
| TigerLLM 1B, LLaMA 3.2 (deprecated) | 0.61 | 0.55 | 0.68 | 0.61 | 0.59 | 0.62 |
| TigerLLM-9B-it (deprecated) | 0.72 | 0.68 | 0.70 | 0.63 | 0.65 | 0.68 |
None of these models were re-run on the five datasets above. The 9B and 12B checkpoints do
not fit a free T4 in 16-bit, and quantising them would have produced a fourth
incomparable measurement rather than a comparison. Anyone with the hardware can settle it:
the notebook that produced this card takes a model id and the same five datasets.
### Sample generations
| Prompt | `gemma-3-1b-it` | This model |
|---|---|---|
| বাংলাদেশের মুক্তিযুদ্ধ সম্পর্কে তিনটি বাক্যে লিখুন। | অবশ্যই! বাংলাদেশের মুক্তিযুদ্ধের ইতিহাস নিচে তিনটি বাক্যে তুলে ধরা হলো: বাংলাদেশের মুক্তিযুদ্ধ | বাংলাদেশের মুক্তিযুদ্ধ ১৯৫২ থেকে ১৯৬৪ সালের মধ্যে সংঘটিত হয়, যখন দেশের স্বাধীনতা সংগ্রামের জন্ |
| রবীন্দ্রনাথ ঠাকুর কে ছিলেন? | রবীন্দ্রনাথ ঠাকুর ছিলেন একজন বিখ্যাত বাঙালি কবি, ঔপন্যাসিক, নাট্যকার, দার্শনিক, সংগীতজ্ঞ, এবং স | রবীন্দ্রনাথ ঠাকুর (১৮৬১-১৯৪১) ছিলেন একটি বিখ্যাত ভারতীয় কবি, নাট্যকার, উপন্যাসিক, দার্শনিক এবং |
| একটি ছোট গল্প লিখুন যেখানে একজন কৃষক বৃষ্টির জন্য অপেক্ষা | বৃষ্টির জন্য অপেক্ষা করা একজন কৃষকের গল্প: সকালটা শুরু হয়েছিল পাখির কলকাকলি আর ধুলোর আঘাতেই। রফ | একটি ছোট গ্রামে ছিল একটি কৃষক, নাম ছিল রহিম। তিনি তার জমি থেকে সবজি ও ফল উৎপাদন করতেন এবং মাঝে |
| পানি ফুটলে কী ঘটে, সহজ ভাষায় ব্যাখ্যা করুন। | পানি ফুটলে যা ঘটে, তা হলো: ১. তাপ দেওয়া: যখন পানি ফুটিয়ে নিলে, সেটির তাপমাত্রা বেড়ে যায়। এই তাপ | পানি ফুটলে কী ঘটে, তা বুঝতে হলে আমাদের কিছু মৌলিক পদার্থবিজ্ঞান ধারণা বুঝতে হবে। ### ধাপ ১: তাপ |
These are greedy decodes, and they are in the card because the benchmark tables cannot
answer the question they answer. Read them before trusting any number above.
## Tokenizer efficiency on Bangla
| | tokens per word |
|---|---|
| Bangla | 1.42 |
| English | 1.30 |
Gemma-3's 262k-token vocabulary covers 140+ languages, and Bangla costs **1.1×** as
many tokens per word as English here. Budget context length accordingly.
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tok = AutoTokenizer.from_pretrained("emonHasan01/BanglaGPT-Pro")
model = AutoModelForCausalLM.from_pretrained("emonHasan01/BanglaGPT-Pro", torch_dtype=torch.float16,
device_map="auto")
messages = [
{"role": "system", "content": "আপনি একজন সহায়ক সহকারী। সবসময় স্পষ্ট ও সঠিক বাংলায় উত্তর দিন।"},
{"role": "user", "content": "বাংলাদেশের মুক্তিযুদ্ধ সম্পর্কে তিনটি বাক্যে লিখুন।"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
The Bangla system prompt was present in every training example and is in
[`prompt_format.json`](prompt_format.json). Results degrade without it.
### llama.cpp
| File | Size |
|---|---|
| `BanglaGPT-Pro.Q4_K_M.gguf` | 0.81 GB |
| `BanglaGPT-Pro.Q8_0.gguf` | 1.07 GB |
```bash
llama-cli -hf emonHasan01/BanglaGPT-Pro-GGUF:Q4_K_M -p "রবীন্দ্রনাথ ঠাকুর কে ছিলেন?" -n 256
```
## Training
| | |
|---|---|
| Base model | `google/gemma-3-1b-it` |
| Method | **full fine-tune**, every parameter, via Unsloth |
| Dataset | `Bangla-Instruct`, 342,391 rows available |
| Examples trained on | **80,000**, one pass, no repeats |
| Steps | 5,000 at batch 2 × 8 accumulation = 16 effective, length-grouped |
| Max sequence length | 512 tokens; longer examples dropped, not truncated |
| Learning rate | 1e-5, cosine schedule |
| Optimizer | `adamw_8bit`, `max_grad_norm=1.0` |
| Hardware | Tesla T4, free Kaggle tier, across multiple sessions with checkpoint-resume |
### Decisions that mattered
**Full fine-tuning, and why it fits.** Gemma-3-1B is roughly 2 GB in fp16 and gradients add
2 GB. Plain AdamW would need about 8 GB for its fp32 moment buffers and would not fit a T4;
`adamw_8bit` brings that to around 2 GB, and gradient checkpointing absorbs the
activations. At 3B this would not fit without quantising the base, at which point it is
QLoRA, not a full fine-tune.
**Learning rate 1e-5, not 2e-4.** Every weight moves here, so steps must be small. LoRA
tolerates 2e-4 because a fraction of a percent of parameters carry the update.
**Native Bangla data, not translated Alpaca.** Most Bangla fine-tunes use machine-translated
Alpaca derivatives, which carry English sentence structure into the model. `Bangla-Instruct`
was written natively: 50 volunteers wrote seed tasks, then GPT-4 and Claude 3.5 Sonnet
expanded them with self-instruct.
**Loss on the assistant turn only.** Prompt tokens are masked to `-100`.
## Limitations
- **1B parameters.** Fluent Bangla is not accurate Bangla. In the sample table above this
model answers in clean Bangla and still dates the Liberation War wrong. Perplexity says
it learned the language; that answer says it did not learn the facts.
- **No continual pretraining.** Instruction tuning adapts form, not knowledge. This is the
main gap between this model and TituLLM or TigerLLM, both of which pretrained on Bangla
text first.
- **Trained on 512-token examples.** Nothing breaks past that — Gemma-3 handles
32k — but the fine-tune contributes nothing beyond it and the model prefers answers of
roughly that length.
- **The benchmarks barely separate anything** at this scale, as the chance column shows.
- **No safety, bias or faithfulness evaluation**, in Bangla or English.
- **English ability was not measured** after fine-tuning. Hishab measured this for their
model and found large English regressions after Bangla-only training; expect the same
here, unquantified.
- **Religious, political and regional sensitivity** was not evaluated.
## What a v2 would change
1. **Continual pretraining on Bangla text before instruction tuning.** The largest lever
by far, and the one every stronger Bangla model used: TituLLM 37B tokens, BongLLaMA
6.8B, TigerLLM 10M from school textbooks. This model used none.
2. **A larger base.** At 4B, factual capacity stops being the binding constraint, which is
what the Bangla MMLU column is actually measuring.
3. **More of the dataset.** 80,000 of 342,391 rows were used.
4. **Benchmarks that discriminate at this scale**, plus systematic human reading of
generations, which is what matters for a writing assistant.
## Reproducing
Two Kaggle notebooks on a Tesla T4: one trained the model, one produced every number,
chart and table on this page. Raw numbers, timings and the scorer's self-check are in
[`results.json`](results.json).
## Citation
```bibtex
@inproceedings{raihan-zampieri-2025-tigerllm,
title = {{TigerLLM} - A Family of {B}angla Large Language Models},
author = {Raihan, Nishat and Zampieri, Marcos},
booktitle = {Proceedings of ACL},
year = {2025}
}
@inproceedings{nahin-etal-2025-titullms,
title = {{TituLLMs}: A Family of {B}angla {LLM}s with Comprehensive Benchmarking},
author = {Nahin, Shahriar Kabir and Nandi, Rabindra Nath and Sarker, Sagor and others},
booktitle = {Findings of ACL},
year = {2025}
}
@software{unsloth,
title = {Unsloth}, author = {Daniel Han and Michael Han and Unsloth team},
year = {2023}, url = {https://github.com/unslothai/unsloth}
}
```