| --- |
| language: |
| - en |
| tags: |
| - nanochat |
| --- |
| |
| # Small language models trained with nanochat (Assignment 1, Agentic LLMs, Leiden University) |
|
|
| Small language models trained with [nanochat](https://github.com/karpathy/nanochat) on one NVIDIA RTX 2070 SUPER |
| (8 GB), for Assignment 1 of the Agentic LLMs course. Authors: Kevin Bretz and An Nguyen. |
| Code, setup and all commands: https://github.com/kevin-bretz/Small-Language-Model |
|
|
| | Folder | Model | Training | |
| |---|---|---| |
| | `checkpoints/d2_pretraining` | depth 2, 13.0M parameters | pre-training on ClimbMix, 420 steps, 55M tokens | |
| | `checkpoints/d4_pretraining` | depth 4, 36.7M parameters | pre-training on ClimbMix, 528 steps, 138M tokens | |
| | `checkpoints/d2_mid`, `checkpoints/d4_mid` | | stage 1, mid-training on MMLU + GSM8K, 310 steps | |
| | `checkpoints/d2_sft`, `checkpoints/d4_sft` | | stage 2, SFT on SmolTalk, 500 steps | |
| | `tokenizer` | BPE tokenizer with 32,768 tokens, used by all models | 500M characters of ClimbMix | |
| | `tokenizer_8192` | BPE tokenizer with 8,192 tokens (tokenizer comparison only) | 500M characters of ClimbMix | |
|
|
| Every checkpoint folder contains the weights (`model_*.pt`), nanochat's metadata (`meta_*.json`) and the optimizer |
| state (`optim_*_rank0.pt`). |
|
|
| ## Using the checkpoints |
|
|
| 1. Set up the code and the virtual environment as described in the README of our |
| [GitHub repository](https://github.com/kevin-bretz/Small-Language-Model). |
| 2. Copy the folders of this repository into nanochat's cache folder `~/.cache/nanochat/`: |
|
|
| | This repository | `~/.cache/nanochat/` | |
| |---|---| |
| | `checkpoints/d2_pretraining`, `checkpoints/d4_pretraining` | `base_checkpoints/d2`, `base_checkpoints/d4` | |
| | `checkpoints/d2_mid`, `d2_sft`, `d4_mid`, `d4_sft` | `chatsft_checkpoints/d2_mid`, `d2_sft`, `d4_mid`, `d4_sft` | |
| | `tokenizer` | `tokenizer` and `tokenizer_32768` (the same files) | |
| | `tokenizer_8192` | `tokenizer_8192` | |
|
|
| 3. From the repository root, run for example: |
|
|
| python -m assignment.task3 eval d4 # ARC-Easy, ARC-Challenge and GSM8K after each training stage |
| python -m assignment.task4 d4_sft # our temperature study through nanochat's scripts/chat_cli.py |
| |
| or chat with a model yourself: `python -m scripts.chat_cli -i sft -g d4_sft` |
|
|
