| --- |
| license: mit |
| language: |
| - gu |
| - ne |
| library_name: pytorch |
| tags: |
| - transformer |
| - language-model |
| - from-scratch |
| --- |
| |
| # LMA mini project - monolingual Transformer LMs for Gujarati and Nepali |
|
|
| Ten checkpoints from five independently pretrained decoder-only Transformers, |
| written from scratch in PyTorch (no `nn.Transformer*`, no HuggingFace model |
| classes, no pre-built attention). Each model is ~25M parameters, trained for |
| one epoch over its own monolingual corpus. |
|
|
| Every run stores two checkpoints: `best.pt` (lowest validation loss) and |
| `latest.pt` (end of the run). Both are resume-capable - they carry model |
| weights, optimizer state, scheduler state, gradient-scaler state, step, token |
| count, both configs and the RNG states. |
|
|
| ## The five runs |
|
|
| | Folder | Language | Vocab | Layers | Positional | Params | Steps | Test PPL | Test BPB | What it is | |
| |---|---|---:|---:|---|---:|---:|---:|---:|---| |
| | `gujarati-model-h/16k-rope/` | Gujarati (Model H) | 16,000 | 11 | rope | 25,625,856 | 15,826 | 51.62 | 0.5764 | reported model | |
| | `gujarati-model-h/12k-rope/` | Gujarati (Model H) | 12,000 | 12 | rope | 25,860,864 | 16,418 | 44.64 | 0.5759 | vocabulary control | |
| | `gujarati-model-h/16k-nopos/` | Gujarati (Model H) | 16,000 | 11 | none | 25,625,856 | 15,826 | 56.07 | 0.5885 | bonus ablation, no positional encoding | |
| | `nepali-model-l/16k-rope/` | Nepali (Model L) | 16,000 | 11 | rope | 25,625,856 | 14,962 | 38.93 | 0.4247 | reported model | |
| | `nepali-model-l/12k-rope/` | Nepali (Model L) | 12,000 | 12 | rope | 25,860,864 | 15,564 | 33.47 | 0.4236 | vocabulary control | |
|
|
| **Perplexity is only comparable within a vocabulary.** The 12k runs look |
| better on perplexity purely because choosing 1 of 12,000 pieces is an easier |
| question than 1 of 16,000. Bits-per-byte divides by bytes of text instead, so |
| it is the number to compare across tokenizers. |
|
|
| ## Loading a checkpoint |
|
|
| ```python |
| import torch |
| blob = torch.load('best.pt', map_location='cpu', weights_only=False) |
| print(blob['model_config']) # architecture that produced these weights |
| print(blob['state']['step']) # training step |
| # rebuild with the DecoderLM class from the project repository |
| ``` |
|
|
| The model code, tokenizers, configs, training logs and full evaluation are in |
| the project repository; these files are only the weights. |
|
|
