File size: 2,277 Bytes
c925d55
 
 
 
 
 
 
 
 
b319e33
 
 
 
6a3b7e4
 
 
 
 
 
 
 
 
 
 
 
 
b319e33
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42e588e
 
b319e33
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
---
library_name: pytorch
pipeline_tag: text-generation
tags:
  - gpt
  - from-scratch
  - pretraining
---

# algomaster99/pretrain-2.2e18

Pretrain run `pretrain_2.2e18_bf16_qknorm` (see `src/model/train.py`).

## Summary

A decoder-only GPT-style causal language model, pretrained from scratch (no pretrained-weight initialization) on Spanish, Portuguese, and Hindi text. Model shape (172M params) and training token budget were sized to a 2.2e18 FLOPs compute budget via an IsoFLOP scaling-law fit (`scripts/fit_isoflop_scaling_law.py`). Trained in bf16 autocast with QK-LayerNorm bounding the attention logits -- earlier bf16 attempts without it diverged from unbounded attention-logit growth (see `scripts/slurm/11_pretrain_2.2e18_qknorm.sh`).

## Data

Pretrained on [`andre15silva/pretrain-pt-es-hi`](https://huggingface.co/datasets/andre15silva/pretrain-pt-es-hi), a cross-source MinHash-deduped corpus built from:

- Spanish, Portuguese: `HuggingFaceFW/fineweb-2` (es/pt)
- Hindi: `HuggingFaceFW/fineweb-2` (hi) + `ai4bharat/sangraha`'s "verified" split (human-verified sites, OCR'd PDFs, transcribed speech -- the one Hindi source that isn't itself another Common-Crawl derivative)

No language/quality filtering beyond that dedup is applied -- sources are used as-is (see the repo's own README for the full pipeline). Tokenized with a shared multilingual byte-level BPE tokenizer (vocab_size 32000) trained on the same corpus.

## Model config

```
GPTConfig(vocab_size=32000, block_size=1024, n_layer=11, n_head=16, n_embd=1024, dropout=0.0, bias=True, qk_norm=True)
```

## Final training state

- iter: 259653
- final_train_loss: 3.1813
- final_val_loss: 2.6673

## Files

- `ckpt_final.pt` -- final trained weights (`model_state_dict`, `model_cfg`, `iter_num`, `final_train_loss`, `final_val_loss`)
- `ckpt.pt` -- best-val-loss checkpoint seen during training
- `ckpt_iter*.pt` -- periodic snapshots (no optimizer state)

`ckpt_last.pt` (optimizer state, for resuming training) is intentionally **not** published here -- it's redundant with the files above for anyone just loading weights, and adds ~2GB per snapshot for state nobody outside the original training job needs.

Load with `torch.load(path, weights_only=False)` and `src.model.gpt.GPTConfig`/`GPT`.