mvGPT checkpoints
The models the YouTube series Minimum Viable GPT trains on camera, as
mvGPT's own checkpoints: each model.pt holds the model's config, its weights and its tokenizer (a
vocabulary or BPE merges), and loads with load_checkpoint from
mvGPT. They're the same files as in the videos.
The series' Colab notebooks, in the code's notebooks/, download them, so an episode can start from
the models the episodes before it trained. For transformers and Transformers.js, the model is at
nenuadrian/mvGPT.
The files sit where mvGPT's scripts look for them, so in a clone of the code:
from huggingface_hub import hf_hub_download
hf_hub_download("nenuadrian/mvGPT-checkpoints", "checkpoints/sft/model.pt", local_dir=".")
| File | Episode | The model |
|---|---|---|
checkpoints/pretrain/model.pt |
2 | 14.6M parameters, pretrained on TinyStories for 10 minutes on a laptop, with word-level tokens |
checkpoints/sft/model.pt |
3 | fine-tuned on TinyStories-Instruct for 10 minutes |
checkpoints/dpo/model.pt |
7 | the SFT model after 10 minutes of DPO |
checkpoints/ppo/kl-0.05/model.pt, kl-0.5/model.pt |
8 | the SFT model after 10 minutes of PPO, with a KL weight of 0.05 and of 0.5 |
checkpoints/lora/model.pt, adapters.pt |
9 | the pretrained model fine-tuned with LoRA for 10 minutes, merged, and the adapters alone |
checkpoints/pretrain-rope/model.pt |
10 | pretrained with rotary position embeddings |
checkpoints/pretrain-rope-bpe/model.pt |
11 | pretrained with the C++ BPE tokenizer and rotary embeddings |
checkpoints/pretrain-ddp/model.pt |
12 | 91.3M parameters, pretrained on all of TinyStories for 30 minutes on 4 RTX 4090s |
checkpoints/pretrain-rope-bpe-diffusion/model.pt |
15 | a masked diffusion model, 10 minutes on a laptop |
checkpoints/pretrain-rope-bpe-diffusion-converted/model.pt |
15 | a masked diffusion model from the next-token model of episode 11, 5 minutes |
checkpoints/diffusion-ddp/model.pt |
15 | a masked diffusion model, 30 minutes on 4 RTX 4090s |
checkpoints/diffusion-ddp-converted/model.pt |
15 | a masked diffusion model from pretrain-ddp, 15 minutes on 4 RTX 4090s |
checkpoints/pretrain-web/model.pt |
19 | 110.2M parameters, pretrained on FineWeb-Edu for a night on one RTX 4090 |
checkpoints/tools/model.pt |
20 | the web model fine-tuned to call a calculator |
checkpoints/agent/model.pt |
21 | the web model fine-tuned into an agent's three roles |
checkpoints/long-pi/model.pt, long-yarn/model.pt |
22 | the web model fine-tuned at 4,096 tokens, stretched with PI and with YaRN |
results/bakeoff/*.json |
23 | the optimizer race's runs, which bakeoff_report.py reads |
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support