metadata
license: mit
LM optimization: checkpoints
Checkpoints from https://github.com/LamShiuChing/LM-optimization, a from-scratch study of what text a small model should think in and learn from. Every run is a 5M-parameter GPT (6 layers, d=256, no positional encoding, attention cut at document boundaries) trained on TinyStories prose mixed with generated reasoning documents.
One folder per run: ckpt.pt (state dict, loadable with src/model.py from the repo), config.json
(all hyperparameters and the dataset version), log.jsonl (loss and task accuracies during training),
summary.json (final accuracies). Findings F1-F21 are in the repo's docs/findings.md.