| license: mit | |
| # LM optimization: checkpoints | |
| Checkpoints from https://github.com/LamShiuChing/LM-optimization, a from-scratch study of what text a small | |
| model should think in and learn from. Every run is a 5M-parameter GPT (6 layers, d=256, no positional encoding, | |
| attention cut at document boundaries) trained on TinyStories prose mixed with generated reasoning documents. | |
| One folder per run: `ckpt.pt` (state dict, loadable with `src/model.py` from the repo), `config.json` | |
| (all hyperparameters and the dataset version), `log.jsonl` (loss and task accuracies during training), | |
| `summary.json` (final accuracies). Findings F1-F21 are in the repo's `docs/findings.md`. | |