firstlight-124m / README.md
adarsh-007's picture
Card: link the GitHub code
8799056 verified
|
Raw History Blame Contribute Delete
8.11 kB
---
license: mit
library_name: pytorch
pipeline_tag: text-generation
language:
- en
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smol-smoltalk
tags:
- gpt2
- from-scratch
- pretraining
- sft
- chat
- nanochat
---
# Firstlight-124M
A 124M-parameter language model **pretrained from scratch** on 9.83B tokens of FineWeb-EDU, then turned into a
**chat assistant** with supervised fine-tuning (SFT) on smol-smoltalk. GPT-2 architecture, written in plain PyTorch.
Two models in this repo:
- **`base/`**: the pretrained model. It continues text.
- **`chat/`**: the base model after SFT. It answers a question and stops.
**Try it:** [Firstlight Chat](https://adarsheluri143--firstlight-chat-ui.us-east.modal.run) (the full model, hosted on
Modal; the first visit after a quiet spell takes ~30 s to start). The "Base vs Chat" tab sends the same prompt to both
models.
**Code:** [github.com/adarsh070502/firstlight](https://github.com/adarsh070502/firstlight) (pretraining, SFT, evals,
release and demo).
| Model | HellaSwag (acc_norm) |
|---|---|
| OpenAI GPT-2 124M | 29.55% |
| **Firstlight-124M base** | **30.30%** |
| **Firstlight-124M chat** | 29.81% |
## Model details
| | |
|---|---|
| Developed by | Adarsh ([adarsh-007](https://huggingface.co/adarsh-007)) |
| Type | decoder-only transformer (GPT-2): 12 layers, 12 heads, 768 dim, 1024-token context |
| Parameters | 124,475,904 |
| Tokenizer | GPT-2 BPE (`tiktoken`) + 4 chat tokens |
| Language | English |
| License | MIT |
## Uses
- **Direct use:** learning and demos: how pretraining and SFT change a model's behaviour, small-model experiments,
a starting point for preference tuning (DPO) or LoRA experiments.
- **Out of scope:** anything that needs correct facts or safe output. It has no preference tuning and no safety
training.
## How to use
The network, chat format and sampling (with a KV cache) are in [`model.py`](model.py) in this repo
(needs `torch`, `tiktoken`, `safetensors`, `huggingface_hub`):
```python
import shutil
from huggingface_hub import hf_hub_download
shutil.copy(hf_hub_download("adarsh-007/firstlight-124m", "model.py"), "model.py")
from model import load_model, render_chat, stream, enc
chat = load_model("chat")
ids = render_chat([{"role": "user", "content": "What is photosynthesis?"}])
reply = ""
for reply in stream(chat, ids, max_new_tokens=300, temperature=0.7, top_k=50):
pass
print(reply)
base = load_model("base") # continues plain text
text = ""
for text in stream(base, enc.encode("The best way to learn a language is"), chat=False):
pass
print(text)
```
**Chat format:**
```
<|endoftext|> <|user_start|> What is 2+2? <|user_end|> <|assistant_start|> It is 4. <|assistant_end|>
```
The four chat tokens use ids 50257–50260, unused slots of the vocabulary (padded from 50,257 to 50,304).
## Limitations
At 124M parameters the chat model gets the **format** right (direct answers, numbered lists, **bold** headers,
stopping by itself) but not the **content**: it states wrong facts confidently ("Paris ... about 2,500 miles south
of the City of Light"), miscounts (asked for 3 tips, gives 5 or 8) and repeats itself in long answers. SFT teaches
behaviour, not knowledge.
Multi-turn chat works ("Can you make it shorter?" rewrites its previous answer), but **small talk** is weak: a bare
"thanks" often gets the topic repeated instead of "you're welcome". smol-smoltalk is almost all task conversations,
so the model rarely saw that kind of turn.
Both models see at most **1,024 tokens** (about 750 words). In long chats the oldest turns are forgotten.
## Training
### Pretraining (base model)
- **Data:** [FineWeb-EDU](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) `sample-10BT`,
**9.83B tokens** (1 epoch).
- **Recipe:** 18,751 steps × 0.5M tokens, AdamW, LR 6e-4 with warmup and cosine decay, bf16.
- **Result:** val loss **3.0436**; HellaSwag 30.30% vs 29.55% for OpenAI's GPT-2 124M.
<details>
<summary>Hyperparameters</summary>
| | |
|---|---|
| Steps | 18,751 × 524,288 tokens (2^19) |
| Micro-batch | 4 × 1024 tokens, 128 gradient-accumulation steps per optimizer step |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on 2-D weights, grad clip 1.0 |
| Learning rate | linear warmup 715 steps to 6e-4, cosine decay to 6e-5 |
| Precision | bf16 autocast, TF32 matmuls, `torch.compile`, flash attention |
| Throughput | ~16.4K tokens/s, ~7 days |
| Validation | first 100M-token shard held out; best val loss at step 18,250 |
| Checkpoints | every 100 steps (atomic write); resumed twice on the exact next tokens |
</details>
### SFT (chat model)
- **Data:** [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), **403,748
conversations** (253M tokens), a dataset made for models under 1B.
- **Loss:** the same next-token loss as pretraining, **only on the assistant's tokens and `<|assistant_end|>`**
(so it learns to stop); user tokens and padding are ignored.
- **Recipe:** 1 epoch, 12,617 steps × 32 conversations, LR 5e-5, fresh AdamW, 5.8 hours.
- **Result:** val loss on assistant tokens **2.478 → 1.4495**, with no overfitting.
![SFT loss](sft_loss.png)
<details>
<summary>Data filtering</summary>
| | Train | Test |
|---|---|---|
| raw conversations | 460,341 | 24,229 |
| dropped: Python coding (`self-oss-instruct`); FineWeb-EDU has almost no code | 48,071 | 2,590 |
| dropped: the first answer alone is longer than 1,024 tokens | 8,522 | 448 |
| cut after the last complete assistant turn that fits in 1,024 tokens | 178,102 | 9,429 |
| **kept** | **403,748** | **21,191** |
| tokens | 253M (81% trained on) | 13M |
</details>
<details>
<summary>Hyperparameters</summary>
| | |
|---|---|
| Start | base model weights, a fresh AdamW |
| Batch | 32 conversations per step (micro-batch 4 × 8 accumulation), one conversation per row, right-padded |
| Learning rate | 5e-5, warmup 50 steps, linear decay to 0; no weight decay |
| Steps | 12,617 = 1 epoch |
| Padding | conversations sorted by length inside each step: padding waste 29% → 6% |
| Throughput | ~12.2K tokens/s (no `torch.compile`: every batch has a different length) |
| Validation | masked loss on the same 500 test conversations every 250 steps |
</details>
## Evaluation
**HellaSwag**, all 10,042 validation examples, completion style as in build-nanogpt (each ending is scored by its
average token loss after the context; the lowest wins; random = 25%). Same script for every row:
| Model | acc | acc_norm |
|---|---|---|
| OpenAI GPT-2 124M | 28.58% | 29.55% |
| **Firstlight-124M base** | **28.86%** | **30.30%** |
| Firstlight-124M chat | - | 29.81% |
SFT cost 0.5 points: the chat model kept most of the pretraining knowledge.
**Same question, before and after SFT:**
```
Q: Hi! Can you give me three tips for studying for an exam?
base: =/?src=?src=?src=?src=?src=?src=?src=?src=... (in the chat format: never stops)
chat: 1. **Evaluate your answers**: Make a note of your answers to the exam questions. ...
2. **Take your time**: Make a plan for studying. ...
3. **Use study materials**: Choose study materials that are relevant to your studies ...
```
The chat model stopped by itself on 27 of 50 test questions within 300 tokens (real answers in the data are long
too: median 239 tokens).
## Files
| File | Contents |
|---|---|
| `base/model.safetensors` | pretrained model, fp32 |
| `chat/model.safetensors` | SFT model, fp32 |
| `config.json` | model size + special token ids |
| `model.py` | the network, chat format and sampling |
| `sft_loss.png` | the SFT loss plot |
`lm_head.weight` is not stored: it is tied to `transformer.wte.weight`, and the loader re-ties it.
## Credits and license
- Andrej Karpathy
- Data: FineWeb-EDU (ODC-By 1.0) and smol-smoltalk (Apache-2.0), both by Hugging Face.
- Weights and code: MIT.