Initial release: OdiaGPT-30M trained from scratch

README.md

@@ -0,0 +1,76 @@
+---
+language:
+- or
+license: apache-2.0
+tags:
+- odia
+- odiagpt
+- transformer-from-scratch
+- indic
+- causal-lm
+pipeline_tag: text-generation
+widget:
+- text: "ଓଡ଼ିଶାର ଲୋକମାନେ"
+- text: "ଆଜି ସକାଳେ"
+- text: "ଓଡ଼ିଆ ଭାଷାର"
+---
+
+# OdiaGPT-30M (Pretrained from Scratch)
+
+**OdiaGPT-30M** is an autoregressive decoder-only Transformer built and trained **completely from scratch** for the Odia language.
+
+It does **not** use pretrained GPT-2, Llama, or Indic model weights, nor off-the-shelf tokenizers. All 31.5 million parameters began from random Gaussian initialization, and its 8,192-token BPE tokenizer was trained exclusively on a cleaned subset of the AI4Bharat Sangraha Odia corpus.
+
+## Model Summary
+- **Organization / Creator**: [CodeHima](https://huggingface.co/CodeHima)
+- **Architecture**: Decoder-only Transformer (Llama-compatible)
+- **Parameters**: 31,465,984 (~31.5M)
+- **Layers**: 8
+- **Hidden dimension**: 512 (8 attention heads, head dimension: 64)
+- **Feed-Forward**: SwiGLU ($d_{ff} = 1536$)
+- **Positional Encoding**: Rotary Position Embeddings (RoPE, base 10000.0)
+- **Normalization**: RMSNorm ($\epsilon = 10^{-6}$)
+- **Context Length**: 512
+- **Vocabulary**: 8,192 (SentencePiece BPE with byte fallback)
+- **Precision**: FP16 (Safetensors format)
+
+## Training Dynamics
+- **Dataset**: AI4Bharat Sangraha (`verified/ori`) — 28.5M tokens / 110M cleaned characters
+- **Tokens Seen**: ~29.5 Million tokens (1,800 steps)
+- **Initial Loss**: 9.07 -> **Final Validation Loss**: 4.664 (Perplexity: **106.09**)
+- **Hardware**: Single NVIDIA Tesla T4 on Google Colab (~35,200 tokens/sec)
+
+## Usage with Hugging Face Transformers
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "CodeHima/OdiaGPT-30M"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16).cuda()
+
+prompt = "ଓଡ଼ିଶାର ଲୋକମାନେ"
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=64,
+    temperature=0.8,
+    top_k=50,
+    top_p=0.9,
+    repetition_penalty=1.15,
+    do_sample=True
+)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Sample Output
+> **Prompt**: `ଆଜି ସକାଳେ`  
+> **Completion**: *ଆଜି ସକାଳେ ସକାଳେ ଦୁଇ ଝିଅ ସହ ଆସି ଶୋଇପଡ଼ିଲା । ସେହି ସମୟରେ ବାପା ଘର ଲୋକ କୁ ଖବର ଦେଇ କହିଥିଲେ ଯେ, "ତେବେ ମାଆ, ତମେ ଆଉ କିଛି କହିଦେବୁ । ମୁଁ ଏ କଥା କିଛି ଶୁଣିନାହିଁ ।"*
+
+## Limitations
+- Base foundation model, not instruction-tuned.
+- Developed as part of the OdiaGPT from-scratch educational curriculum.

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "dtype": "float16",
+  "eos_token_id": 3,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "max_position_embeddings": 512,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
+  "use_cache": true,
+  "vocab_size": 8192
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.16.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a00fe5d9d493cfb7b01f8bfea5e0d31f3debad2805eace598c9b523272ec101f
+size 62940032

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26b8026425d1e4f58f9acc6c7334968de2885988a824c005574e2baa079e6dfb
+size 191665

tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>",
+  "clean_up_tokenization_spaces": false,
+  "model_max_length": 512,
+  "tokenizer_class": "LlamaTokenizer"
+}