File size: 677 Bytes
a166c82 | 1 2 3 4 5 6 7 8 9 10 11 12 13 | ---
license: mit
---
# LM optimization: checkpoints
Checkpoints from https://github.com/LamShiuChing/LM-optimization, a from-scratch study of what text a small
model should think in and learn from. Every run is a 5M-parameter GPT (6 layers, d=256, no positional encoding,
attention cut at document boundaries) trained on TinyStories prose mixed with generated reasoning documents.
One folder per run: `ckpt.pt` (state dict, loadable with `src/model.py` from the repo), `config.json`
(all hyperparameters and the dataset version), `log.jsonl` (loss and task accuracies during training),
`summary.json` (final accuracies). Findings F1-F21 are in the repo's `docs/findings.md`.
|