OdiaSLM-100M-Base: Native Odia Small Language Model (100,685,568 (100.69M) Parameters)

OdiaSLM-100M-Base is a from-scratch decoder-only Small Language Model (SLM) pre-trained specifically for the Odia language (or, ଓଡ଼ିଆ) on 1.57 Billion tokens drawn from a strictly deduplicated and benchmark-decontaminated corpus of 9,619,050 clean documents (1.247 Billion words).

Built on a modern Llama-style architecture (Grouped-Query Attention, SwiGLU, Half-Split RoPE, Pre-Norm FP32-Variance RMSNorm, Tied Input/Output Embeddings, and Auxiliary Z-Loss), OdiaSLM-100M-Base loads natively in Hugging Face transformers (LlamaForCausalLM), vLLM, TGI, and llama.cpp with trust_remote_code=False.

Official Research Paper (DOI): 10.5281/zenodo.24167433 | Download Full PDF Paper (OdiaSLM_Preprint_Misra_2026.pdf) | GitHub Repository


1. Model Architecture Specification

Hyperparameter Value
Model Variant OdiaSLM-100M-Base
Architecture Class LlamaForCausalLM (trust_remote_code=False)
Total Trainable Parameters 100,685,568 (100.69M) (Tied Embeddings)
Transformer Layers ($L$) 14
Hidden Dimension ($d_{model}$) 768
Attention Heads ($H_Q$ / $H_{KV}$) 12 Query / 4 KV (GQA)
Head Dimension ($d_{head}$) 64 (Half-Split RoPE, $\theta = 10,000$)
SwiGLU Intermediate Size ($d_{ff}$) 2048
Vocabulary Size ($V$) 16,384 (Odia SentencePiece BPE + Byte Fallback)
Tokenizer Fertility (Odia) 1.4809 tokens/word (11.5727 UTF-8 bytes/token, 0 <unk>)
Context Window 1,024 tokens (~690 Odia words)
Pretraining Tokens Processed 1,572,864,000 (1.57B)
Best Held-Out Validation Loss (CE) 2.8417

2. Training Telemetry & Convergence Figures

Loss and BPB Curve Perplexity and Next-Token Accuracy WSD Schedule and Grad Norm Layer-Wise Gradient Flow Heatmap Weight RMS and Logit Stability Hardware Throughput and VRAM


3. Intrinsic Evaluation: Domain-Stratified Perplexity & Bits-Per-Byte (BPB)

Validation Domain Cross-Entropy (CE) Perplexity (PPL) Bits-Per-Byte (BPB) Top-1 Token Acc Top-5 Token Acc
sangraha:verified_ori 3.0754 21.66 0.3750 40.46% 62.87%
v2:odia_web_corpus_v5 2.9469 19.05 0.3531 42.42% 64.94%
v2:varta 2.8242 16.85 0.3238 43.86% 66.74%
v2:indiccorp_v2 3.0930 22.04 0.3841 40.94% 62.43%
v2:indic_nlp_corp 2.9978 20.04 0.3916 43.17% 63.34%
anchor:fineweb_edu_en 3.0579 21.28 1.3085 41.20% 62.13%

Domain BPB and Token Accuracy Positional Context Scaling


4. Zero-Shot Downstream Evaluation (saidutta69/odia-eval-benchmark)

  • Pure-Odia 18-Task Macro Average: 60.83% (PMI = 59.86%, +34.44% lift over 26.39% random baseline)
  • All 27 Sub-Benchmarks Macro Average: 56.83% (+30.01% lift over 26.82% random baseline)
Benchmark Task Script Mode Random Chance Raw Acc Byte-Norm Acc PMI Calibrated Acc Lift Over Chance
indic_xparaphrase (classification) pure_odia 25.0% 93.8% 95.0% 100.0% +75.0%
flores_plus (translation) cross_lingual_en_choices 25.0% 36.2% 37.5% 96.2% +71.2%
indic_headline_gen (generation) pure_odia 25.0% 53.8% 57.5% 93.8% +68.8%
indic_qa (qa) pure_odia 25.0% 73.8% 55.0% 93.8% +68.8%
odia_gsm8k (math) pure_odia 25.0% 63.8% 32.5% 93.8% +68.8%
indic_quest_odia (qa) pure_odia 25.0% 27.5% 38.8% 87.5% +62.5%
indic_sentence_summarization (generation) pure_odia 25.0% 51.2% 51.2% 86.2% +61.2%
odia_truthfulqa (qa) pure_odia 25.0% 46.2% 62.5% 86.2% +61.2%
indic_question_generation (generation) pure_odia 25.0% 62.5% 57.5% 83.8% +58.8%
bhram_il (qa) cross_lingual_en_choices 25.0% 32.5% 38.8% 71.2% +46.2%
sarvam_gsm8k_indic (math) cross_lingual_en_choices 25.0% 36.2% 33.8% 68.8% +43.8%
indic_squad_odia (qa) pure_odia 25.0% 27.5% 28.8% 53.8% +28.8%
indic_glue_wstp (multiple_choice) pure_odia 25.0% 43.8% 28.8% 46.2% +21.2%
odia_truthfulqa_mc (multiple_choice) cross_lingual_en_choices 24.1% 31.2% 36.2% 30.0% +12.1%
indic_copa (multiple_choice) pure_odia 50.0% 57.5% 58.8% 46.2% +8.8%

Downstream Benchmark Bars Tokenizer Fertility Comparison Corpus Composition and Decontamination


5. Quickstart: Python (transformers + sentencepiece)

import sentencepiece as spm
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM

repo_id = "arunjmisra2273/OdiaSLM-100M-Base"
tok_path = hf_hub_download(repo_id=repo_id, filename="tokenizer.model")
sp = spm.SentencePieceProcessor(model_file=tok_path)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    trust_remote_code=False,
)
model.eval()

prompt = "ଓଡ଼ିଶାର ରାଜଧାନୀ ଭୁବନେଶ୍ୱର ଏକ"
input_ids = torch.tensor([[1] + sp.encode(prompt, out_type=int)], dtype=torch.long)

with torch.no_grad():
    out_ids = model.generate(
        input_ids,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=2,
        pad_token_id=0,
    )

print(sp.decode([int(t) for t in out_ids[0].tolist() if int(t) not in (0, 1, 2)]))

6. Dual License (CC-BY-NC-4.0 + Custom Commercial License)

  • Code: Licensed under Apache License 2.0.
  • Model Weights & Tokenizer: Licensed under CC-BY-NC-4.0 for academic, research, and non-commercial use. Commercial deployments require a separate Commercial License from the author (arunjmisra2273). See LICENSE.md for details.

7. Citation (BibTeX)

@article{misra2026odiaslm,
  title={OdiaSLM: From-Scratch Pretraining, Morphology-Aware Tokenization, and Calibrated Evaluation of 60M and 100M Small Language Models for Odia},
  author={Misra, Arun},
  year={2026},
  month={November},
  publisher={Zenodo},
  doi={10.5281/zenodo.24167433},
  url={https://doi.org/10.5281/zenodo.24167433}
}
Downloads last month
59
Safetensors
Model size
0.1B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for arunjmisra2273/OdiaSLM-100M-Base

Finetunes
1 model
Quantizations
1 model

Space using arunjmisra2273/OdiaSLM-100M-Base 1