File size: 2,277 Bytes
c925d55 b319e33 6a3b7e4 b319e33 42e588e b319e33 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ---
library_name: pytorch
pipeline_tag: text-generation
tags:
- gpt
- from-scratch
- pretraining
---
# algomaster99/pretrain-2.2e18
Pretrain run `pretrain_2.2e18_bf16_qknorm` (see `src/model/train.py`).
## Summary
A decoder-only GPT-style causal language model, pretrained from scratch (no pretrained-weight initialization) on Spanish, Portuguese, and Hindi text. Model shape (172M params) and training token budget were sized to a 2.2e18 FLOPs compute budget via an IsoFLOP scaling-law fit (`scripts/fit_isoflop_scaling_law.py`). Trained in bf16 autocast with QK-LayerNorm bounding the attention logits -- earlier bf16 attempts without it diverged from unbounded attention-logit growth (see `scripts/slurm/11_pretrain_2.2e18_qknorm.sh`).
## Data
Pretrained on [`andre15silva/pretrain-pt-es-hi`](https://huggingface.co/datasets/andre15silva/pretrain-pt-es-hi), a cross-source MinHash-deduped corpus built from:
- Spanish, Portuguese: `HuggingFaceFW/fineweb-2` (es/pt)
- Hindi: `HuggingFaceFW/fineweb-2` (hi) + `ai4bharat/sangraha`'s "verified" split (human-verified sites, OCR'd PDFs, transcribed speech -- the one Hindi source that isn't itself another Common-Crawl derivative)
No language/quality filtering beyond that dedup is applied -- sources are used as-is (see the repo's own README for the full pipeline). Tokenized with a shared multilingual byte-level BPE tokenizer (vocab_size 32000) trained on the same corpus.
## Model config
```
GPTConfig(vocab_size=32000, block_size=1024, n_layer=11, n_head=16, n_embd=1024, dropout=0.0, bias=True, qk_norm=True)
```
## Final training state
- iter: 259653
- final_train_loss: 3.1813
- final_val_loss: 2.6673
## Files
- `ckpt_final.pt` -- final trained weights (`model_state_dict`, `model_cfg`, `iter_num`, `final_train_loss`, `final_val_loss`)
- `ckpt.pt` -- best-val-loss checkpoint seen during training
- `ckpt_iter*.pt` -- periodic snapshots (no optimizer state)
`ckpt_last.pt` (optimizer state, for resuming training) is intentionally **not** published here -- it's redundant with the files above for anyone just loading weights, and adds ~2GB per snapshot for state nobody outside the original training job needs.
Load with `torch.load(path, weights_only=False)` and `src.model.gpt.GPTConfig`/`GPT`.
|