--- license: mit library_name: pytorch pipeline_tag: text-generation language: - en datasets: - HuggingFaceFW/fineweb-edu - HuggingFaceTB/smol-smoltalk tags: - gpt2 - from-scratch - pretraining - sft - chat - nanochat --- # Firstlight-124M A 124M-parameter language model **pretrained from scratch** on 9.83B tokens of FineWeb-EDU, then turned into a **chat assistant** with supervised fine-tuning (SFT) on smol-smoltalk. GPT-2 architecture, written in plain PyTorch. Two models in this repo: - **`base/`**: the pretrained model. It continues text. - **`chat/`**: the base model after SFT. It answers a question and stops. **Try it:** [Firstlight Chat](https://adarsheluri143--firstlight-chat-ui.us-east.modal.run) (the full model, hosted on Modal; the first visit after a quiet spell takes ~30 s to start). The "Base vs Chat" tab sends the same prompt to both models. **Code:** [github.com/adarsh070502/firstlight](https://github.com/adarsh070502/firstlight) (pretraining, SFT, evals, release and demo). | Model | HellaSwag (acc_norm) | |---|---| | OpenAI GPT-2 124M | 29.55% | | **Firstlight-124M base** | **30.30%** | | **Firstlight-124M chat** | 29.81% | ## Model details | | | |---|---| | Developed by | Adarsh ([adarsh-007](https://huggingface.co/adarsh-007)) | | Type | decoder-only transformer (GPT-2): 12 layers, 12 heads, 768 dim, 1024-token context | | Parameters | 124,475,904 | | Tokenizer | GPT-2 BPE (`tiktoken`) + 4 chat tokens | | Language | English | | License | MIT | ## Uses - **Direct use:** learning and demos: how pretraining and SFT change a model's behaviour, small-model experiments, a starting point for preference tuning (DPO) or LoRA experiments. - **Out of scope:** anything that needs correct facts or safe output. It has no preference tuning and no safety training. ## How to use The network, chat format and sampling (with a KV cache) are in [`model.py`](model.py) in this repo (needs `torch`, `tiktoken`, `safetensors`, `huggingface_hub`): ```python import shutil from huggingface_hub import hf_hub_download shutil.copy(hf_hub_download("adarsh-007/firstlight-124m", "model.py"), "model.py") from model import load_model, render_chat, stream, enc chat = load_model("chat") ids = render_chat([{"role": "user", "content": "What is photosynthesis?"}]) reply = "" for reply in stream(chat, ids, max_new_tokens=300, temperature=0.7, top_k=50): pass print(reply) base = load_model("base") # continues plain text text = "" for text in stream(base, enc.encode("The best way to learn a language is"), chat=False): pass print(text) ``` **Chat format:** ``` <|endoftext|> <|user_start|> What is 2+2? <|user_end|> <|assistant_start|> It is 4. <|assistant_end|> ``` The four chat tokens use ids 50257–50260, unused slots of the vocabulary (padded from 50,257 to 50,304). ## Limitations At 124M parameters the chat model gets the **format** right (direct answers, numbered lists, **bold** headers, stopping by itself) but not the **content**: it states wrong facts confidently ("Paris ... about 2,500 miles south of the City of Light"), miscounts (asked for 3 tips, gives 5 or 8) and repeats itself in long answers. SFT teaches behaviour, not knowledge. Multi-turn chat works ("Can you make it shorter?" rewrites its previous answer), but **small talk** is weak: a bare "thanks" often gets the topic repeated instead of "you're welcome". smol-smoltalk is almost all task conversations, so the model rarely saw that kind of turn. Both models see at most **1,024 tokens** (about 750 words). In long chats the oldest turns are forgotten. ## Training ### Pretraining (base model) - **Data:** [FineWeb-EDU](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample-10BT`, **9.83B tokens** (1 epoch). - **Recipe:** 18,751 steps × 0.5M tokens, AdamW, LR 6e-4 with warmup and cosine decay, bf16. - **Result:** val loss **3.0436**; HellaSwag 30.30% vs 29.55% for OpenAI's GPT-2 124M.
Hyperparameters | | | |---|---| | Steps | 18,751 × 524,288 tokens (2^19) | | Micro-batch | 4 × 1024 tokens, 128 gradient-accumulation steps per optimizer step | | Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on 2-D weights, grad clip 1.0 | | Learning rate | linear warmup 715 steps to 6e-4, cosine decay to 6e-5 | | Precision | bf16 autocast, TF32 matmuls, `torch.compile`, flash attention | | Throughput | ~16.4K tokens/s, ~7 days | | Validation | first 100M-token shard held out; best val loss at step 18,250 | | Checkpoints | every 100 steps (atomic write); resumed twice on the exact next tokens |
### SFT (chat model) - **Data:** [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), **403,748 conversations** (253M tokens), a dataset made for models under 1B. - **Loss:** the same next-token loss as pretraining, **only on the assistant's tokens and `<|assistant_end|>`** (so it learns to stop); user tokens and padding are ignored. - **Recipe:** 1 epoch, 12,617 steps × 32 conversations, LR 5e-5, fresh AdamW, 5.8 hours. - **Result:** val loss on assistant tokens **2.478 → 1.4495**, with no overfitting. ![SFT loss](sft_loss.png)
Data filtering | | Train | Test | |---|---|---| | raw conversations | 460,341 | 24,229 | | dropped: Python coding (`self-oss-instruct`); FineWeb-EDU has almost no code | 48,071 | 2,590 | | dropped: the first answer alone is longer than 1,024 tokens | 8,522 | 448 | | cut after the last complete assistant turn that fits in 1,024 tokens | 178,102 | 9,429 | | **kept** | **403,748** | **21,191** | | tokens | 253M (81% trained on) | 13M |
Hyperparameters | | | |---|---| | Start | base model weights, a fresh AdamW | | Batch | 32 conversations per step (micro-batch 4 × 8 accumulation), one conversation per row, right-padded | | Learning rate | 5e-5, warmup 50 steps, linear decay to 0; no weight decay | | Steps | 12,617 = 1 epoch | | Padding | conversations sorted by length inside each step: padding waste 29% → 6% | | Throughput | ~12.2K tokens/s (no `torch.compile`: every batch has a different length) | | Validation | masked loss on the same 500 test conversations every 250 steps |
## Evaluation **HellaSwag**, all 10,042 validation examples, completion style as in build-nanogpt (each ending is scored by its average token loss after the context; the lowest wins; random = 25%). Same script for every row: | Model | acc | acc_norm | |---|---|---| | OpenAI GPT-2 124M | 28.58% | 29.55% | | **Firstlight-124M base** | **28.86%** | **30.30%** | | Firstlight-124M chat | - | 29.81% | SFT cost 0.5 points: the chat model kept most of the pretraining knowledge. **Same question, before and after SFT:** ``` Q: Hi! Can you give me three tips for studying for an exam? base: =/?src=?src=?src=?src=?src=?src=?src=?src=... (in the chat format: never stops) chat: 1. **Evaluate your answers**: Make a note of your answers to the exam questions. ... 2. **Take your time**: Make a plan for studying. ... 3. **Use study materials**: Choose study materials that are relevant to your studies ... ``` The chat model stopped by itself on 27 of 50 test questions within 300 tokens (real answers in the data are long too: median 239 tokens). ## Files | File | Contents | |---|---| | `base/model.safetensors` | pretrained model, fp32 | | `chat/model.safetensors` | SFT model, fp32 | | `config.json` | model size + special token ids | | `model.py` | the network, chat format and sampling | | `sft_loss.png` | the SFT loss plot | `lm_head.weight` is not stored: it is tied to `transformer.wte.weight`, and the loader re-ties it. ## Credits and license - Andrej Karpathy - Data: FineWeb-EDU (ODC-By 1.0) and smol-smoltalk (Apache-2.0), both by Hugging Face. - Weights and code: MIT.