|
Download README.md from algomaster99/pretrain-2.2e18: direct link, hf CLI and curl.
- Browser
- Download file 2.28 kB
-
https://huggingface.co/algomaster99/pretrain-2.2e18/resolve/main/README.md
- Command line
-
hf download hf://algomaster99/pretrain-2.2e18/README.md
-
curl -L -o README.md https://huggingface.co/algomaster99/pretrain-2.2e18/resolve/main/README.md
2.28 kB
| library_name: pytorch | |
| pipeline_tag: text-generation | |
| tags: | |
| - gpt | |
| - from-scratch | |
| - pretraining | |
| # algomaster99/pretrain-2.2e18 | |
| Pretrain run `pretrain_2.2e18_bf16_qknorm` (see `src/model/train.py`). | |
| ## Summary | |
| A decoder-only GPT-style causal language model, pretrained from scratch (no pretrained-weight initialization) on Spanish, Portuguese, and Hindi text. Model shape (172M params) and training token budget were sized to a 2.2e18 FLOPs compute budget via an IsoFLOP scaling-law fit (`scripts/fit_isoflop_scaling_law.py`). Trained in bf16 autocast with QK-LayerNorm bounding the attention logits -- earlier bf16 attempts without it diverged from unbounded attention-logit growth (see `scripts/slurm/11_pretrain_2.2e18_qknorm.sh`). | |
| ## Data | |
| Pretrained on [`andre15silva/pretrain-pt-es-hi`](https://huggingface.co/datasets/andre15silva/pretrain-pt-es-hi), a cross-source MinHash-deduped corpus built from: | |
| - Spanish, Portuguese: `HuggingFaceFW/fineweb-2` (es/pt) | |
| - Hindi: `HuggingFaceFW/fineweb-2` (hi) + `ai4bharat/sangraha`'s "verified" split (human-verified sites, OCR'd PDFs, transcribed speech -- the one Hindi source that isn't itself another Common-Crawl derivative) | |
| No language/quality filtering beyond that dedup is applied -- sources are used as-is (see the repo's own README for the full pipeline). Tokenized with a shared multilingual byte-level BPE tokenizer (vocab_size 32000) trained on the same corpus. | |
| ## Model config | |
| ``` | |
| GPTConfig(vocab_size=32000, block_size=1024, n_layer=11, n_head=16, n_embd=1024, dropout=0.0, bias=True, qk_norm=True) | |
| ``` | |
| ## Final training state | |
| - iter: 259653 | |
| - final_train_loss: 3.1813 | |
| - final_val_loss: 2.6673 | |
| ## Files | |
| - `ckpt_final.pt` -- final trained weights (`model_state_dict`, `model_cfg`, `iter_num`, `final_train_loss`, `final_val_loss`) | |
| - `ckpt.pt` -- best-val-loss checkpoint seen during training | |
| - `ckpt_iter*.pt` -- periodic snapshots (no optimizer state) | |
| `ckpt_last.pt` (optimizer state, for resuming training) is intentionally **not** published here -- it's redundant with the files above for anyone just loading weights, and adds ~2GB per snapshot for state nobody outside the original training job needs. | |
| Load with `torch.load(path, weights_only=False)` and `src.model.gpt.GPTConfig`/`GPT`. | |