| --- |
| license: mit |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - nanochat |
| - gpt |
| - from-scratch |
| - chat |
| - reinforcement-learning |
| - grpo |
| - mechanistic-interpretability |
| - llm |
| datasets: |
| - nvidia/Nemotron-ClimbMix |
| --- |
| |
| # WillyGPT |
|
|
| A 1.68B-parameter chat model trained from scratch β pretraining, supervised finetuning (SFT), and GRPO reinforcement learning β on the [nanochat](https://github.com/karpathy/nanochat) recipe (MIT). Built & trained by William Yates on a single 8ΓH100 node. |
|
|
| Base pretraining clears the GPT-2 CORE reference (0.3016 vs 0.2565). Two checkpoints are released: an **SFT generalist (default)** and an **RL math-specialist**. |
|
|
| - **Full recipe, training scripts, and reproduction:** [github.com/williamyates/WillyGPT](https://github.com/williamyates/WillyGPT) |
| - **Architecture:** nanochat `d26` β these weights are *not* Hugging Face Transformers and will not load with `AutoModel.from_pretrained`. Run them with the nanochat code (below). |
|
|
| ## Files in this repo |
|
|
| | Path | Checkpoint | Use | |
| |---|---|---| |
| | `chatsft_checkpoints/d26/model_000483.pt` | **SFT** (default) | Best all-rounder; chat, identity, spelling | |
| | `chatrl_checkpoints/d26/model_000466.pt` | **RL** | GSM8K-specialized; see decoding note below | |
| | `tokenizer/` | `tokenizer.pkl`, `token_bytes.pt` | 32,768-token BPE | |
|
|
| ## Download & run |
|
|
| ```bash |
| # 1. get nanochat |
| git clone https://github.com/karpathy/nanochat.git && cd nanochat |
| uv sync --extra gpu # or: uv sync --extra cpu (Mac/MPS, slow but works) |
| |
| # 2. download these weights into nanochat's cache (structure is preserved) |
| export NANOCHAT_BASE_DIR=$PWD |
| hf download williamyates/WillyGPT --local-dir "$NANOCHAT_BASE_DIR/.cache/nanochat" |
| |
| # 3. chat |
| python -m scripts.chat_cli -i sft -p "Who are you?" # SFT: generalist (recommended) |
| python -m scripts.chat_cli -i rl -p "What is 17 * 23?" # RL: math specialist |
| python -m scripts.chat_web # browser UI |
| ``` |
|
|
| A 1.68B model runs (slowly) on Apple Silicon via MPS, or on any single modern GPU. |
|
|
| **Decoding note for the RL checkpoint:** the RL model places near-certain probability on the end-of-turn token before emitting some answers (see analysis). If it terminates early, suppress `<|assistant_end|>` for the first ~80 generated tokens (a `min_new_tokens` constraint) β this restores SpellingBee from ~18% to ~95% with no weight changes. The SFT checkpoint does not need this. |
|
|
| ## Model specification |
|
|
| | | | |
| |---|---| |
| | Architecture | GPT-style decoder transformer (nanochat `d26`) | |
| | Parameters | 1.68B (189 weight tensors) | |
| | Layers / heads | 26 layers Β· 13 attention heads Β· 13 KV heads Β· head dim 128 | |
| | Model dimension | 1664 | |
| | Context length | 2048 | |
| | Attention | Sliding-window (SSSL), Flash Attention 3 | |
| | Precision | fp8 training (Hopper / H100) | |
| | Tokenizer | 32,768-token BPE, GPT-4 style | |
| | Training data | NVIDIA ClimbMix (pretrain); SmolTalk + MMLU + GSM8K + custom identity set (SFT); GSM8K (RL) | |
|
|
| ## Evaluation |
|
|
| Base on the pretraining CORE benchmark; SFT and RL on the nanochat chat suite. Accuracies (0β1) except the CORE/ChatCORE composites. |
|
|
| | Metric | Base | SFT | RL | Ξ SFTβRL | |
| |---|---:|---:|---:|---:| |
| | CORE (pretrain) | 0.3016 | β | β | β | |
| | ChatCORE | β | 0.3982 | 0.2639 | β13.4 pt | |
| | GSM8K | β | 0.1099 | 0.1729 | +6.3 pt | |
| | SpellingBee | β | 0.9961 | 0.1836 | β81.3 pt | |
| | HumanEval | β | 0.1524 | 0.0915 | β6.1 pt | |
| | MMLU | β | 0.3831 | 0.3797 | β0.3 pt | |
| | ARC-Easy | β | 0.6843 | 0.6911 | +0.7 pt | |
| | ARC-Challenge | β | 0.5307 | 0.5307 | Β±0.0 | |
|
|
| CORE 0.3016 exceeds the GPT-2 reference of 0.2565. **The SFT checkpoint is the default**; RL is a math-specialized variant. |
|
|
| ### External comparison (directional β methodologies differ) |
|
|
| | Model | CORE | MMLU | ARC-E | ARC-C | GSM8K | |
| |---|---:|---:|---:|---:|---:| |
| | WillyGPT Β· 1.68B Β· '26 | 0.30 | 38.3 | 68.4 | 53.1 | 11.0β17.3 | |
| | GPT-2 XL Β· 1.5B Β· '19 | ~0.26 | ~26 | β | ~30 | ~0 | |
| | nanochat d20 ($100) Β· 561M | 0.22 | 31.5 | 38.8 | 28.1 | 4.6β7.6 | |
| | Qwen2.5-1.5B Β· '24 | β | 59.8 | 79.1 | 53.4 | 68.5 | |
| | SmolLM2-1.7B Β· '24 | β | 51.9 | 77.8 | 50.3 | 47.7 | |
|
|
| At identical parameter count WillyGPT is well ahead of GPT-2 XL; the gap to 2024 1.5B models is specific, not global (ARC-Challenge is level with Qwen2.5-1.5B; the shortfall is concentrated in MMLU and GSM8K β the axes that scale most directly with token budget. Qwen2.5 trained on ~18T tokens vs WillyGPT's 11B). |
|
|
| ## SFT vs. RL analysis (summary) |
|
|
| GRPO improved its single optimization target (GSM8K, +6.3 pt / +57% relative) and left the rest flat or lower β a specialization tax. A logit-level investigation locates a large part of the regression in *when the model stops generating*, not in the underlying skills: |
|
|
| - **SpellingBee is masked.** Raw spelling is identical between SFT and RL (23/24). RL changed termination: `P(<|assistant_end|>)` at the answer position goes 0.000 β 0.999. Suppressing that token for ~80 decode steps recovers ~95% accuracy, inference-only. |
| - **HumanEval is mixed** β roughly half harness/extraction artifact, half real RL mode-collapse (degenerate repetition). |
| - **Identity robustness is thin.** Both checkpoints answer "Who are you?" correctly cold; mid-conversation the RL checkpoint can confabulate a false (DeepMind/AlphaGo) origin, while SFT holds. Mitigate with a system prompt. |
| - **Smear (falsified hypothesis)** An initial weight-diff hypothesis (RL altered "smear" parameters) was falsified by a dose-response ablation β largest relative weight change β functional importance. |
|
|
| Full mechanistic writeup and recommendations: [the GitHub README](https://github.com/williamyates/WillyGPT). |
|
|
| ## Limitations |
|
|
| This is a sub-2B model trained on 11B tokens. It is **not a frontier model** and **confidently hallucinates specific facts** (e.g. inventing place names). The trained-in honesty about being unreliable does not make the outputs reliable β treat all specific factual claims as unverified. Knowledge breadth (MMLU) and multi-step math (GSM8K) are the weakest areas, as expected from the token budget. |
|
|
| ## Provenance |
|
|
| Both checkpoints were verified bit-for-bit (SHA-256) between the GPU node and a laptop before teardown, deserialize cleanly on CPU/MPS (189 tensors, ~1.68B params), and run locally on a 32 GB M4 MacBook Air via MPS. |
|
|
| ## Credits & license |
|
|
| MIT-licensed; a derivative of [nanochat](https://github.com/karpathy/nanochat) by Andrej Karpathy (also MIT). The nanochat framework β architecture, training scripts, tokenizer, and eval harness β is Karpathy's work. This release adds the d26 configuration, the identity layer, the GRPO stage, and the accompanying analysis. Pretraining data: NVIDIA ClimbMix. SFT data: SmolTalk, MMLU, GSM8K, plus a custom identity set. |
