MizGPT

MizGPT is a causal language model fine-tuned for the Mizo language (Lushai). It is based on openai-community/gpt2 (~124M parameters) and trained on a curated corpus of cleaned and tokenized Mizo text.

Developed and maintained by Hillbyte.

---

Model Details

• Model Name: MizGPT
• Model ID: hillbyte/MizGPT
• Architecture: GPT-2 (Causal Language Model / GPT2LMHeadModel)
• Base Model: openai-community/gpt2
• Parameters: ~124 Million
• Primary Language: Mizo (lus) / English
• License: MIT
• Training Corpus: Curated Mizo text corpus (125,341 training samples)

---

Evaluation & Performance

The model was evaluated on a dedicated validation split (7,403 samples) over 15 epochs (156,690 optimization steps):

Metric	Score
Perplexity	24.99
Validation Accuracy	39.18%
Validation Loss	3.2185
Training Loss	1.4742
Training Epochs	15.0
Total FLOPs	9.83e+17

---

How to Use

1. Using Hugging Face Pipeline

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="hillbyte/MizGPT",
    device=-1, # use 0 for CUDA
)

prompt = "Kan khawvel hi"
output = generator(
    prompt,
    max_new_tokens=50,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True,
)

print(output[0]["generated_text"])

2. Using PyTorch & AutoModelForCausalLM

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "hillbyte/MizGPT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Set padding token
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

prompt = "Thawnthu: "
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output_tokens = model.generate(
        **inputs,
        max_new_tokens=60,
        temperature=0.7,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.1,
        do_sample=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_tokens[0], skip_special_tokens=True))

---

Recommended Generation Parameters

Because GPT-2 is a 124M parameter model, setting a slight repetition penalty and top-p sampling yields natural and fluent Mizo completions:

• temperature: 0.7
• top_p: 0.9
• top_k: 50
• repetition_penalty: 1.1

---

Intended Uses & Limitations

• Intended Uses: Mizo story generation, creative writing assistance, text completion, and linguistic NLP research in low-resource Indian languages.
• Limitations: Like all autoregressive models, generations should be reviewed for factual correctness and coherence over very long outputs.