MaithiliCLM: First Causal Language Model for the Maithili Language

MaithiliCLM is a decoder-only causal language model (109.5M parameters) pretrained from scratch on MaithiliCorpus — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the first causal/generative language model ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.

Unlike encoder-only models (BERT, RoBERTa) that only support understanding tasks, MaithiliCLM enables text generation, making it suitable for creative writing, dialogue systems, summarization, and other generative applications in Maithili.

Model Architecture

Property	Value
Architecture	Decoder-only Transformer (GPT-family)
Parameters	109,545,984
Vocabulary	Byte-level BPE, 32,000 tokens
Context Window	1,024 tokens
Hidden Size	768
Intermediate Size	3,072
Attention Heads	12
Hidden Layers	12
Activation	SwiGLU
Normalization	RMSNorm
Position Encoding	Rotary Position Embeddings (RoPE)
Dropout	0.1
Training Data	MaithiliCorpus LID-filtered (48,064 documents, 27.5M words)
Training Tokens	983M
Training Steps	30,000
Effective Batch Size	32
Optimizer	AdamW (lr=3e-4, weight_decay=0.1)
LR Schedule	Cosine with warmup
Precision	FP16 mixed
License	CC-BY-4.0

Architecture Details

MaithiliCLM implements a modernized GPT-2 family architecture with the following enhancements:

Rotary Position Embeddings (RoPE): Enables better length generalization than learned absolute positional embeddings
RMSNorm: More efficient than LayerNorm, used in LLaMA and recent models
SwiGLU: Improved activation function over GeLU, used in PaLM and LLaMA
Causal Self-Attention: Standard autoregressive masking for language modeling

Tokenizer

The tokenizer is a byte-level BPE trained from scratch on MaithiliCorpus with a vocabulary of 32,000 tokens. This compact vocabulary reduces embedding parameters from 153.6M to 24.6M, shrinking the embedding layer from 55% to 22% of total model parameters.

Property	Value
Algorithm	Byte-level BPE
Vocabulary Size	32,000
Training Data	MaithiliCorpus (39.8M words)
Tokens per Word	3.53
Special Tokens	`<s>`, `</s>`, `<unk>`, `<pad>`

Training

Data Preparation

The training corpus underwent rigorous cleaning and quality filtering:

Source: AiventraLab/maithili-corpus (49,629 raw documents)
Cleaning: HTML removal, encoding normalization, language verification
LID Filtering: Retained documents with ≥50% Maithili sentences (48,064 documents)
Train/Val Split: 47,103 training / 961 validation documents
Tokenization: Byte-level BPE encoding, packed into 1,024-token blocks

Training Configuration

Duration: ~12 hours
Tokens Seen: 983M (~10.3 epochs)
Peak VRAM: 6.46 GB
Throughput: ~26,500 tokens/second

Training Dynamics

Metric	Step 0	Step 10K	Step 20K	Step 30K
Train Loss	10.32	3.65	3.18	3.04
Val Loss	10.31	3.42	3.18	3.12
Val PPL	29,674	30.56	24.05	22.68

Training plateaued after ~30K steps with diminishing returns (<1 ppl improvement per 5K steps).

Generation Quality

The model generates coherent Maithili text across diverse prompts. Generated continuations are syntactically correct and semantically plausible, though coherence degrades with longer sequences due to the limited training data.

Intended Use

Text Generation: Creative writing, article completion, dialogue systems
Fine-tuning Base: Adapt for classification, NER, QA, summarization
Research: Foundation for Maithili NLP, low-resource language modeling
Education: Teaching Maithili language and computational linguistics

How to Use

from transformers import AutoTokenizer
import torch

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("AiventraLab/maithili-clm")

# Load model (requires custom model class - see model.py)
# model = MaithiliGPT.from_pretrained("AiventraLab/maithili-clm")

# Generate text
prompt = "मैथिली भाषा एक"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Autoregressive generation
output = model.generate(
    input_ids,
    max_new_tokens=100,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    do_sample=True
)

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

Acknowledgments

The authors gratefully acknowledge AnK Komputing for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.

Citation

@article{maithili2026clm,
  title={MaithiliCLM: First Causal Language Model for the Maithili Language},
  author={[Authors]},
  journal={arXiv preprint},
  year={2026}
}

License

CC-BY 4.0 applies to the model weights, tokenizer, and documentation. The training data (MaithiliCorpus) retains its original license terms.

AiventraLab
/

maithili-clm