|
Download README.md from adarsh-007/firstlight-124m: direct link, hf CLI and curl.
- Browser
- Download file 8.11 kB
-
https://huggingface.co/adarsh-007/firstlight-124m/resolve/main/README.md
- Command line
-
hf download hf://adarsh-007/firstlight-124m/README.md
-
curl -L -o README.md https://huggingface.co/adarsh-007/firstlight-124m/resolve/main/README.md
8.11 kB
| license: mit | |
| library_name: pytorch | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| datasets: | |
| - HuggingFaceFW/fineweb-edu | |
| - HuggingFaceTB/smol-smoltalk | |
| tags: | |
| - gpt2 | |
| - from-scratch | |
| - pretraining | |
| - sft | |
| - chat | |
| - nanochat | |
| # Firstlight-124M | |
| A 124M-parameter language model **pretrained from scratch** on 9.83B tokens of FineWeb-EDU, then turned into a | |
| **chat assistant** with supervised fine-tuning (SFT) on smol-smoltalk. GPT-2 architecture, written in plain PyTorch. | |
| Two models in this repo: | |
| - **`base/`**: the pretrained model. It continues text. | |
| - **`chat/`**: the base model after SFT. It answers a question and stops. | |
| **Try it:** [Firstlight Chat](https://adarsheluri143--firstlight-chat-ui.us-east.modal.run) (the full model, hosted on | |
| Modal; the first visit after a quiet spell takes ~30 s to start). The "Base vs Chat" tab sends the same prompt to both | |
| models. | |
| **Code:** [github.com/adarsh070502/firstlight](https://github.com/adarsh070502/firstlight) (pretraining, SFT, evals, | |
| release and demo). | |
| | Model | HellaSwag (acc_norm) | | |
| |---|---| | |
| | OpenAI GPT-2 124M | 29.55% | | |
| | **Firstlight-124M base** | **30.30%** | | |
| | **Firstlight-124M chat** | 29.81% | | |
| ## Model details | |
| | | | | |
| |---|---| | |
| | Developed by | Adarsh ([adarsh-007](https://huggingface.co/adarsh-007)) | | |
| | Type | decoder-only transformer (GPT-2): 12 layers, 12 heads, 768 dim, 1024-token context | | |
| | Parameters | 124,475,904 | | |
| | Tokenizer | GPT-2 BPE (`tiktoken`) + 4 chat tokens | | |
| | Language | English | | |
| | License | MIT | | |
| ## Uses | |
| - **Direct use:** learning and demos: how pretraining and SFT change a model's behaviour, small-model experiments, | |
| a starting point for preference tuning (DPO) or LoRA experiments. | |
| - **Out of scope:** anything that needs correct facts or safe output. It has no preference tuning and no safety | |
| training. | |
| ## How to use | |
| The network, chat format and sampling (with a KV cache) are in [`model.py`](model.py) in this repo | |
| (needs `torch`, `tiktoken`, `safetensors`, `huggingface_hub`): | |
| ```python | |
| import shutil | |
| from huggingface_hub import hf_hub_download | |
| shutil.copy(hf_hub_download("adarsh-007/firstlight-124m", "model.py"), "model.py") | |
| from model import load_model, render_chat, stream, enc | |
| chat = load_model("chat") | |
| ids = render_chat([{"role": "user", "content": "What is photosynthesis?"}]) | |
| reply = "" | |
| for reply in stream(chat, ids, max_new_tokens=300, temperature=0.7, top_k=50): | |
| pass | |
| print(reply) | |
| base = load_model("base") # continues plain text | |
| text = "" | |
| for text in stream(base, enc.encode("The best way to learn a language is"), chat=False): | |
| pass | |
| print(text) | |
| ``` | |
| **Chat format:** | |
| ``` | |
| <|endoftext|> <|user_start|> What is 2+2? <|user_end|> <|assistant_start|> It is 4. <|assistant_end|> | |
| ``` | |
| The four chat tokens use ids 50257–50260, unused slots of the vocabulary (padded from 50,257 to 50,304). | |
| ## Limitations | |
| At 124M parameters the chat model gets the **format** right (direct answers, numbered lists, **bold** headers, | |
| stopping by itself) but not the **content**: it states wrong facts confidently ("Paris ... about 2,500 miles south | |
| of the City of Light"), miscounts (asked for 3 tips, gives 5 or 8) and repeats itself in long answers. SFT teaches | |
| behaviour, not knowledge. | |
| Multi-turn chat works ("Can you make it shorter?" rewrites its previous answer), but **small talk** is weak: a bare | |
| "thanks" often gets the topic repeated instead of "you're welcome". smol-smoltalk is almost all task conversations, | |
| so the model rarely saw that kind of turn. | |
| Both models see at most **1,024 tokens** (about 750 words). In long chats the oldest turns are forgotten. | |
| ## Training | |
| ### Pretraining (base model) | |
| - **Data:** [FineWeb-EDU](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample-10BT`, | |
| **9.83B tokens** (1 epoch). | |
| - **Recipe:** 18,751 steps × 0.5M tokens, AdamW, LR 6e-4 with warmup and cosine decay, bf16. | |
| - **Result:** val loss **3.0436**; HellaSwag 30.30% vs 29.55% for OpenAI's GPT-2 124M. | |
| <details> | |
| <summary>Hyperparameters</summary> | |
| | | | | |
| |---|---| | |
| | Steps | 18,751 × 524,288 tokens (2^19) | | |
| | Micro-batch | 4 × 1024 tokens, 128 gradient-accumulation steps per optimizer step | | |
| | Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on 2-D weights, grad clip 1.0 | | |
| | Learning rate | linear warmup 715 steps to 6e-4, cosine decay to 6e-5 | | |
| | Precision | bf16 autocast, TF32 matmuls, `torch.compile`, flash attention | | |
| | Throughput | ~16.4K tokens/s, ~7 days | | |
| | Validation | first 100M-token shard held out; best val loss at step 18,250 | | |
| | Checkpoints | every 100 steps (atomic write); resumed twice on the exact next tokens | | |
| </details> | |
| ### SFT (chat model) | |
| - **Data:** [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), **403,748 | |
| conversations** (253M tokens), a dataset made for models under 1B. | |
| - **Loss:** the same next-token loss as pretraining, **only on the assistant's tokens and `<|assistant_end|>`** | |
| (so it learns to stop); user tokens and padding are ignored. | |
| - **Recipe:** 1 epoch, 12,617 steps × 32 conversations, LR 5e-5, fresh AdamW, 5.8 hours. | |
| - **Result:** val loss on assistant tokens **2.478 → 1.4495**, with no overfitting. | |
|  | |
| <details> | |
| <summary>Data filtering</summary> | |
| | | Train | Test | | |
| |---|---|---| | |
| | raw conversations | 460,341 | 24,229 | | |
| | dropped: Python coding (`self-oss-instruct`); FineWeb-EDU has almost no code | 48,071 | 2,590 | | |
| | dropped: the first answer alone is longer than 1,024 tokens | 8,522 | 448 | | |
| | cut after the last complete assistant turn that fits in 1,024 tokens | 178,102 | 9,429 | | |
| | **kept** | **403,748** | **21,191** | | |
| | tokens | 253M (81% trained on) | 13M | | |
| </details> | |
| <details> | |
| <summary>Hyperparameters</summary> | |
| | | | | |
| |---|---| | |
| | Start | base model weights, a fresh AdamW | | |
| | Batch | 32 conversations per step (micro-batch 4 × 8 accumulation), one conversation per row, right-padded | | |
| | Learning rate | 5e-5, warmup 50 steps, linear decay to 0; no weight decay | | |
| | Steps | 12,617 = 1 epoch | | |
| | Padding | conversations sorted by length inside each step: padding waste 29% → 6% | | |
| | Throughput | ~12.2K tokens/s (no `torch.compile`: every batch has a different length) | | |
| | Validation | masked loss on the same 500 test conversations every 250 steps | | |
| </details> | |
| ## Evaluation | |
| **HellaSwag**, all 10,042 validation examples, completion style as in build-nanogpt (each ending is scored by its | |
| average token loss after the context; the lowest wins; random = 25%). Same script for every row: | |
| | Model | acc | acc_norm | | |
| |---|---|---| | |
| | OpenAI GPT-2 124M | 28.58% | 29.55% | | |
| | **Firstlight-124M base** | **28.86%** | **30.30%** | | |
| | Firstlight-124M chat | - | 29.81% | | |
| SFT cost 0.5 points: the chat model kept most of the pretraining knowledge. | |
| **Same question, before and after SFT:** | |
| ``` | |
| Q: Hi! Can you give me three tips for studying for an exam? | |
| base: =/?src=?src=?src=?src=?src=?src=?src=?src=... (in the chat format: never stops) | |
| chat: 1. **Evaluate your answers**: Make a note of your answers to the exam questions. ... | |
| 2. **Take your time**: Make a plan for studying. ... | |
| 3. **Use study materials**: Choose study materials that are relevant to your studies ... | |
| ``` | |
| The chat model stopped by itself on 27 of 50 test questions within 300 tokens (real answers in the data are long | |
| too: median 239 tokens). | |
| ## Files | |
| | File | Contents | | |
| |---|---| | |
| | `base/model.safetensors` | pretrained model, fp32 | | |
| | `chat/model.safetensors` | SFT model, fp32 | | |
| | `config.json` | model size + special token ids | | |
| | `model.py` | the network, chat format and sampling | | |
| | `sft_loss.png` | the SFT loss plot | | |
| `lm_head.weight` is not stored: it is tied to `transformer.wte.weight`, and the loader re-ties it. | |
| ## Credits and license | |
| - Andrej Karpathy | |
| - Data: FineWeb-EDU (ODC-By 1.0) and smol-smoltalk (Apache-2.0), both by Hugging Face. | |
| - Weights and code: MIT. | |