---
license: mit
language:
- gu
- ne
library_name: pytorch
tags:
- transformer
- language-model
- from-scratch
---

# LMA mini project - monolingual Transformer LMs for Gujarati and Nepali

Ten checkpoints from five independently pretrained decoder-only Transformers,
written from scratch in PyTorch (no `nn.Transformer*`, no HuggingFace model
classes, no pre-built attention). Each model is ~25M parameters, trained for
one epoch over its own monolingual corpus.

Every run stores two checkpoints: `best.pt` (lowest validation loss) and
`latest.pt` (end of the run). Both are resume-capable - they carry model
weights, optimizer state, scheduler state, gradient-scaler state, step, token
count, both configs and the RNG states.

## The five runs

| Folder | Language | Vocab | Layers | Positional | Params | Steps | Test PPL | Test BPB | What it is |
|---|---|---:|---:|---|---:|---:|---:|---:|---|
| `gujarati-model-h/16k-rope/` | Gujarati (Model H) | 16,000 | 11 | rope | 25,625,856 | 15,826 | 51.62 | 0.5764 | reported model |
| `gujarati-model-h/12k-rope/` | Gujarati (Model H) | 12,000 | 12 | rope | 25,860,864 | 16,418 | 44.64 | 0.5759 | vocabulary control |
| `gujarati-model-h/16k-nopos/` | Gujarati (Model H) | 16,000 | 11 | none | 25,625,856 | 15,826 | 56.07 | 0.5885 | bonus ablation, no positional encoding |
| `nepali-model-l/16k-rope/` | Nepali (Model L) | 16,000 | 11 | rope | 25,625,856 | 14,962 | 38.93 | 0.4247 | reported model |
| `nepali-model-l/12k-rope/` | Nepali (Model L) | 12,000 | 12 | rope | 25,860,864 | 15,564 | 33.47 | 0.4236 | vocabulary control |

**Perplexity is only comparable within a vocabulary.** The 12k runs look
better on perplexity purely because choosing 1 of 12,000 pieces is an easier
question than 1 of 16,000. Bits-per-byte divides by bytes of text instead, so
it is the number to compare across tokenizers.

## Loading a checkpoint

```python
import torch
blob = torch.load('best.pt', map_location='cpu', weights_only=False)
print(blob['model_config'])      # architecture that produced these weights
print(blob['state']['step'])     # training step
# rebuild with the DecoderLM class from the project repository
```

The model code, tokenizers, configs, training logs and full evaluation are in
the project repository; these files are only the weights.