OdiaSLM-60M-Base: Native Odia Small Language Model (62,111,360 (62.11M) Parameters)

OdiaSLM-60M-Base is a from-scratch decoder-only Small Language Model (SLM) pre-trained specifically for the Odia language (or, ଓଡ଼ିଆ) on 1.31 Billion tokens drawn from a strictly deduplicated and benchmark-decontaminated corpus of 9,619,050 clean documents (1.247 Billion words).

Built on a modern Llama-style architecture (Grouped-Query Attention, SwiGLU, Half-Split RoPE, Pre-Norm FP32-Variance RMSNorm, Tied Input/Output Embeddings, and Auxiliary Z-Loss), OdiaSLM-60M-Base loads natively in Hugging Face transformers (LlamaForCausalLM), vLLM, TGI, and llama.cpp with trust_remote_code=False.

Official Research Paper (DOI): 10.5281/zenodo.24167433 | Download Full PDF Paper (OdiaSLM_Preprint_Misra_2026.pdf) | GitHub Repository


1. Model Architecture Specification

Hyperparameter Value
Model Variant OdiaSLM-60M-Base
Architecture Class LlamaForCausalLM (trust_remote_code=False)
Total Trainable Parameters 62,111,360 (62.11M) (Tied Embeddings)
Transformer Layers ($L$) 12
Hidden Dimension ($d_{model}$) 640
Attention Heads ($H_Q$ / $H_{KV}$) 10 Query / 2 KV (GQA)
Head Dimension ($d_{head}$) 64 (Half-Split RoPE, $\theta = 10,000$)
SwiGLU Intermediate Size ($d_{ff}$) 1728
Vocabulary Size ($V$) 16,384 (Odia SentencePiece BPE + Byte Fallback)
Tokenizer Fertility (Odia) 1.4809 tokens/word (11.5727 UTF-8 bytes/token, 0 <unk>)
Context Window 1,024 tokens (~690 Odia words)
Pretraining Tokens Processed 1,310,720,000 (1.31B)
Best Held-Out Validation Loss (CE) 2.9917

2. Training Telemetry & Convergence Figures

Loss and BPB Curve Perplexity and Next-Token Accuracy WSD Schedule and Grad Norm Layer-Wise Gradient Flow Heatmap Weight RMS and Logit Stability Hardware Throughput and VRAM


3. Intrinsic Evaluation: Domain-Stratified Perplexity & Bits-Per-Byte (BPB)

Validation Domain Cross-Entropy (CE) Perplexity (PPL) Bits-Per-Byte (BPB) Top-1 Token Acc Top-5 Token Acc
sangraha:verified_ori 3.1949 24.41 0.3896 39.04% 61.33%
v2:odia_web_corpus_v5 3.0710 21.56 0.3680 41.43% 63.60%
v2:varta 2.9628 19.35 0.3397 42.49% 65.03%
v2:indiccorp_v2 3.2271 25.21 0.4008 39.43% 61.36%
v2:indic_nlp_corp 3.1758 23.95 0.4148 41.51% 61.22%
anchor:fineweb_edu_en 3.1872 24.22 1.3639 39.55% 60.53%

Domain BPB and Token Accuracy Positional Context Scaling


4. Zero-Shot Downstream Evaluation (saidutta69/odia-eval-benchmark)

  • Pure-Odia 18-Task Macro Average: 59.65% (PMI = 58.75%, +33.26% lift over 26.39% random baseline)
  • All 27 Sub-Benchmarks Macro Average: 55.26% (+28.44% lift over 26.82% random baseline)
Benchmark Task Script Mode Random Chance Raw Acc Byte-Norm Acc PMI Calibrated Acc Lift Over Chance
indic_xparaphrase (classification) pure_odia 25.0% 92.5% 95.0% 100.0% +75.0%
odia_gsm8k (math) pure_odia 25.0% 58.8% 35.0% 97.5% +72.5%
indic_qa (qa) pure_odia 25.0% 58.8% 50.0% 93.8% +68.8%
flores_plus (translation) cross_lingual_en_choices 25.0% 33.8% 37.5% 90.0% +65.0%
indic_quest_odia (qa) pure_odia 25.0% 27.5% 38.8% 88.8% +63.8%
indic_question_generation (generation) pure_odia 25.0% 60.0% 62.5% 87.5% +62.5%
odia_truthfulqa (qa) pure_odia 25.0% 45.0% 62.5% 87.5% +62.5%
indic_headline_gen (generation) pure_odia 25.0% 51.2% 55.0% 81.2% +56.2%
indic_sentence_summarization (generation) pure_odia 25.0% 45.0% 51.2% 80.0% +55.0%
bhram_il (qa) cross_lingual_en_choices 25.0% 28.8% 35.0% 67.5% +42.5%
sarvam_gsm8k_indic (math) cross_lingual_en_choices 25.0% 32.5% 32.5% 58.8% +33.8%
indic_squad_odia (qa) pure_odia 25.0% 26.2% 31.2% 53.8% +28.8%
indic_glue_wstp (multiple_choice) pure_odia 25.0% 36.2% 22.5% 36.2% +11.2%
sarvam_triviaqa (multiple_choice) pure_odia 25.0% 55.0% 20.0% 36.2% +11.2%
odia_truthfulqa_mc (multiple_choice) cross_lingual_en_choices 24.1% 32.5% 32.5% 23.8% +8.4%

Downstream Benchmark Bars Tokenizer Fertility Comparison Corpus Composition and Decontamination


5. Quickstart: Python (transformers + sentencepiece)

import sentencepiece as spm
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM

repo_id = "arunjmisra2273/OdiaSLM-60M-Base"
tok_path = hf_hub_download(repo_id=repo_id, filename="tokenizer.model")
sp = spm.SentencePieceProcessor(model_file=tok_path)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    trust_remote_code=False,
)
model.eval()

prompt = "ଓଡ଼ିଶାର ରାଜଧାନୀ ଭୁବନେଶ୍ୱର ଏକ"
input_ids = torch.tensor([[1] + sp.encode(prompt, out_type=int)], dtype=torch.long)

with torch.no_grad():
    out_ids = model.generate(
        input_ids,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=2,
        pad_token_id=0,
    )

print(sp.decode([int(t) for t in out_ids[0].tolist() if int(t) not in (0, 1, 2)]))

6. Dual License (CC-BY-NC-4.0 + Custom Commercial License)

  • Code: Licensed under Apache License 2.0.
  • Model Weights & Tokenizer: Licensed under CC-BY-NC-4.0 for academic, research, and non-commercial use. Commercial deployments require a separate Commercial License from the author (arunjmisra2273). See LICENSE.md for details.

7. Citation (BibTeX)

@article{misra2026odiaslm,
  title={OdiaSLM: From-Scratch Pretraining, Morphology-Aware Tokenization, and Calibrated Evaluation of 60M and 100M Small Language Models for Odia},
  author={Misra, Arun},
  year={2026},
  month={November},
  publisher={Zenodo},
  doi={10.5281/zenodo.24167433},
  url={https://doi.org/10.5281/zenodo.24167433}
}
Downloads last month
47
Safetensors
Model size
62.1M params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for arunjmisra2273/OdiaSLM-60M-Base

Quantizations
1 model

Space using arunjmisra2273/OdiaSLM-60M-Base 1