File size: 2,903 Bytes
d2c36d6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
language:
- or
license: apache-2.0
tags:
- odia
- odiagpt
- transformer-from-scratch
- indic
- causal-lm
pipeline_tag: text-generation
widget:
- text: "ଓଡ଼ିଶାର ଲୋକମାନେ"
- text: "ଆଜି ସକାଳେ"
- text: "ଓଡ଼ିଆ ଭାଷାର"
---
# OdiaGPT-30M (Pretrained from Scratch)
**OdiaGPT-30M** is an autoregressive decoder-only Transformer built and trained **completely from scratch** for the Odia language.
It does **not** use pretrained GPT-2, Llama, or Indic model weights, nor off-the-shelf tokenizers. All 31.5 million parameters began from random Gaussian initialization, and its 8,192-token BPE tokenizer was trained exclusively on a cleaned subset of the AI4Bharat Sangraha Odia corpus.
## Model Summary
- **Organization / Creator**: [CodeHima](https://huggingface.co/CodeHima)
- **Architecture**: Decoder-only Transformer (Llama-compatible)
- **Parameters**: 31,465,984 (~31.5M)
- **Layers**: 8
- **Hidden dimension**: 512 (8 attention heads, head dimension: 64)
- **Feed-Forward**: SwiGLU ($d_{ff} = 1536$)
- **Positional Encoding**: Rotary Position Embeddings (RoPE, base 10000.0)
- **Normalization**: RMSNorm ($\epsilon = 10^{-6}$)
- **Context Length**: 512
- **Vocabulary**: 8,192 (SentencePiece BPE with byte fallback)
- **Precision**: FP16 (Safetensors format)
## Training Dynamics
- **Dataset**: AI4Bharat Sangraha (`verified/ori`) — 28.5M tokens / 110M cleaned characters
- **Tokens Seen**: ~29.5 Million tokens (1,800 steps)
- **Initial Loss**: 9.07 -> **Final Validation Loss**: 4.664 (Perplexity: **106.09**)
- **Hardware**: Single NVIDIA Tesla T4 on Google Colab (~35,200 tokens/sec)
## Usage with Hugging Face Transformers
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "CodeHima/OdiaGPT-30M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16).cuda()
prompt = "ଓଡ଼ିଶାର ଲୋକମାନେ"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
**inputs,
max_new_tokens=64,
temperature=0.8,
top_k=50,
top_p=0.9,
repetition_penalty=1.15,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Sample Output
> **Prompt**: `ଆଜି ସକାଳେ`
> **Completion**: *ଆଜି ସକାଳେ ସକାଳେ ଦୁଇ ଝିଅ ସହ ଆସି ଶୋଇପଡ଼ିଲା । ସେହି ସମୟରେ ବାପା ଘର ଲୋକ କୁ ଖବର ଦେଇ କହିଥିଲେ ଯେ, "ତେବେ ମାଆ, ତମେ ଆଉ କିଛି କହିଦେବୁ । ମୁଁ ଏ କଥା କିଛି ଶୁଣିନାହିଁ ।"*
## Limitations
- Base foundation model, not instruction-tuned.
- Developed as part of the OdiaGPT from-scratch educational curriculum.
|