| --- |
| license: cc-by-nc-4.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| base_model: Wasue/WesleyGPT-Base |
| datasets: |
| - karpathy/climbmix-400b-shuffle |
| - HuggingFaceTB/smol-smoltalk |
| - cais/mmlu |
| - openai/gsm8k |
| tags: |
| - nanochat |
| - trained-from-scratch |
| - small-language-model |
| - consumer-gpu |
| --- |
| <!-- Wesley wrote this --> |
|
|
| # WesleyGPT |
|
|
| A 286-million-parameter chat model trained **from scratch on one RTX 3060** in |
| Wesley Mangum's home: about 14 hours of pretraining, then about 5 hours of |
| fine-tuning to hold a conversation. Built on |
| [nanochat](https://github.com/karpathy/nanochat), Andrej Karpathy's open-source |
| project for training small chat models end to end; the training and serving code |
| is at [WasueM/WesleyGPT](https://github.com/WasueM/WesleyGPT). |
|
|
| It is a small model and it sounds like one: fluent, confident, and often wrong. |
| It is a demonstration of what a single consumer GPU can do, not an assistant to |
| rely on. |
|
|
| ## Try it |
|
|
| ```bash |
| git clone https://github.com/WasueM/WesleyGPT && cd WesleyGPT |
| uv sync --extra cpu --extra release |
| uv run python -m wesleygpt.release chat Wasue/WesleyGPT |
| ``` |
|
|
| Or from Python: |
|
|
| ```python |
| from wesleygpt.release import fetch_release, load_release |
| model, tokenizer = load_release(fetch_release("Wasue/WesleyGPT", "wesleygpt-chat")) |
| ``` |
|
|
| It runs on a laptop CPU. It is not a `transformers` model, so `AutoModel` will |
| not load it. |
|
|
| ## Model |
|
|
| | | | |
| |---|---| |
| | Parameters | 286,261,730 | |
| | Layers / width / heads | 12 / 768 / 6 (query and key-value) | |
| | Context | 2,048 tokens, sliding-window attention (3 short-window layers, then 1 full) | |
| | Tokenizer | 32,768-token byte-level BPE, trained by nanochat on the pretraining data | |
| | Precision | bfloat16 weights | |
|
|
| ## Training |
|
|
| | Stage | Data | Compute | |
| |---|---|---| |
| | Pretraining ([WesleyGPT-Base](https://huggingface.co/Wasue/WesleyGPT-Base)) | 1.32 billion tokens of [ClimbMix](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle) (2,520 steps × 524,288 tokens) | ~14 h, 1× RTX 3060 12 GB | |
| | Supervised fine-tuning | [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) conversations, plus [MMLU](https://huggingface.co/datasets/cais/mmlu) (×3) and [GSM8K](https://huggingface.co/datasets/openai/gsm8k) (×4) training sets, plus a small slice of synthetic conversations about its own identity, 934 steps | ~5 h, same GPU | |
|
|
| ## Evaluation |
|
|
| nanochat's chat evaluation, before and after fine-tuning. The multiple-choice |
| tasks have four options, so 25% is chance. |
|
|
| | Task | Base (pretrained only) | WesleyGPT (chat) | |
| |---|---|---| |
| | ARC-Easy | 23.6% | **36.8%** | |
| | ARC-Challenge | 25.1% | **32.3%** | |
| | MMLU | 26.9% | **31.8%** | |
| | GSM8K (grade-school math) | 0.0% | 1.1% | |
| | HumanEval (Python) | 0.0% | 9.2% | |
| | ChatCORE | 0.002 | **0.090** | |
| | Knows its name (12 held-out questions × 5 samples) | – | 28% | |
|
|
| ## Limitations |
|
|
| - **It makes things up.** Asked about Paris, it said the city borders the French |
| Riviera. |
| - **It only sometimes knows its own name.** Fine-tuning on identity |
| conversations raised correct self-identification on unseen phrasings from 0% |
| to 28%; the rest of the time it falls back on generic assistant personas from |
| its chat data. |
| - Arithmetic and multi-step reasoning are close to zero (GSM8K 1.1%). |
| - English only. No safety tuning beyond what the fine-tuning data carries. |
|
|
| ## License |
|
|
| **CC-BY-NC-4.0** for these weights: free to use, share, and adapt with credit, |
| but not commercially. That matches the license of NVIDIA's |
| [Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix), |
| from which the pretraining data is derived. The code is MIT. |
|
|
