Download README.md from adarsh-007/firstlight-124m: direct link, hf CLI and curl.
- Browser
- Download file 8.11 kB
-
https://huggingface.co/adarsh-007/firstlight-124m/resolve/main/README.md
- Command line
-
hf download hf://adarsh-007/firstlight-124m/README.md
-
curl -L -o README.md https://huggingface.co/adarsh-007/firstlight-124m/resolve/main/README.md
license: mit
library_name: pytorch
pipeline_tag: text-generation
language:
- en
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smol-smoltalk
tags:
- gpt2
- from-scratch
- pretraining
- sft
- chat
- nanochat
Firstlight-124M
A 124M-parameter language model pretrained from scratch on 9.83B tokens of FineWeb-EDU, then turned into a chat assistant with supervised fine-tuning (SFT) on smol-smoltalk. GPT-2 architecture, written in plain PyTorch.
Two models in this repo:
base/: the pretrained model. It continues text.chat/: the base model after SFT. It answers a question and stops.
Try it: Firstlight Chat (the full model, hosted on Modal; the first visit after a quiet spell takes ~30 s to start). The "Base vs Chat" tab sends the same prompt to both models.
Code: github.com/adarsh070502/firstlight (pretraining, SFT, evals, release and demo).
| Model | HellaSwag (acc_norm) |
|---|---|
| OpenAI GPT-2 124M | 29.55% |
| Firstlight-124M base | 30.30% |
| Firstlight-124M chat | 29.81% |
Model details
| Developed by | Adarsh (adarsh-007) |
| Type | decoder-only transformer (GPT-2): 12 layers, 12 heads, 768 dim, 1024-token context |
| Parameters | 124,475,904 |
| Tokenizer | GPT-2 BPE (tiktoken) + 4 chat tokens |
| Language | English |
| License | MIT |
Uses
- Direct use: learning and demos: how pretraining and SFT change a model's behaviour, small-model experiments, a starting point for preference tuning (DPO) or LoRA experiments.
- Out of scope: anything that needs correct facts or safe output. It has no preference tuning and no safety training.
How to use
The network, chat format and sampling (with a KV cache) are in model.py in this repo
(needs torch, tiktoken, safetensors, huggingface_hub):
import shutil
from huggingface_hub import hf_hub_download
shutil.copy(hf_hub_download("adarsh-007/firstlight-124m", "model.py"), "model.py")
from model import load_model, render_chat, stream, enc
chat = load_model("chat")
ids = render_chat([{"role": "user", "content": "What is photosynthesis?"}])
reply = ""
for reply in stream(chat, ids, max_new_tokens=300, temperature=0.7, top_k=50):
pass
print(reply)
base = load_model("base") # continues plain text
text = ""
for text in stream(base, enc.encode("The best way to learn a language is"), chat=False):
pass
print(text)
Chat format:
<|endoftext|> <|user_start|> What is 2+2? <|user_end|> <|assistant_start|> It is 4. <|assistant_end|>
The four chat tokens use ids 50257–50260, unused slots of the vocabulary (padded from 50,257 to 50,304).
Limitations
At 124M parameters the chat model gets the format right (direct answers, numbered lists, bold headers, stopping by itself) but not the content: it states wrong facts confidently ("Paris ... about 2,500 miles south of the City of Light"), miscounts (asked for 3 tips, gives 5 or 8) and repeats itself in long answers. SFT teaches behaviour, not knowledge.
Multi-turn chat works ("Can you make it shorter?" rewrites its previous answer), but small talk is weak: a bare "thanks" often gets the topic repeated instead of "you're welcome". smol-smoltalk is almost all task conversations, so the model rarely saw that kind of turn.
Both models see at most 1,024 tokens (about 750 words). In long chats the oldest turns are forgotten.
Training
Pretraining (base model)
- Data: FineWeb-EDU
sample-10BT, 9.83B tokens (1 epoch). - Recipe: 18,751 steps × 0.5M tokens, AdamW, LR 6e-4 with warmup and cosine decay, bf16.
- Result: val loss 3.0436; HellaSwag 30.30% vs 29.55% for OpenAI's GPT-2 124M.
Hyperparameters
| Steps | 18,751 × 524,288 tokens (2^19) |
| Micro-batch | 4 × 1024 tokens, 128 gradient-accumulation steps per optimizer step |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on 2-D weights, grad clip 1.0 |
| Learning rate | linear warmup 715 steps to 6e-4, cosine decay to 6e-5 |
| Precision | bf16 autocast, TF32 matmuls, torch.compile, flash attention |
| Throughput | ~16.4K tokens/s, ~7 days |
| Validation | first 100M-token shard held out; best val loss at step 18,250 |
| Checkpoints | every 100 steps (atomic write); resumed twice on the exact next tokens |
SFT (chat model)
- Data: smol-smoltalk, 403,748 conversations (253M tokens), a dataset made for models under 1B.
- Loss: the same next-token loss as pretraining, only on the assistant's tokens and
<|assistant_end|>(so it learns to stop); user tokens and padding are ignored. - Recipe: 1 epoch, 12,617 steps × 32 conversations, LR 5e-5, fresh AdamW, 5.8 hours.
- Result: val loss on assistant tokens 2.478 → 1.4495, with no overfitting.
Data filtering
| Train | Test | |
|---|---|---|
| raw conversations | 460,341 | 24,229 |
dropped: Python coding (self-oss-instruct); FineWeb-EDU has almost no code |
48,071 | 2,590 |
| dropped: the first answer alone is longer than 1,024 tokens | 8,522 | 448 |
| cut after the last complete assistant turn that fits in 1,024 tokens | 178,102 | 9,429 |
| kept | 403,748 | 21,191 |
| tokens | 253M (81% trained on) | 13M |
Hyperparameters
| Start | base model weights, a fresh AdamW |
| Batch | 32 conversations per step (micro-batch 4 × 8 accumulation), one conversation per row, right-padded |
| Learning rate | 5e-5, warmup 50 steps, linear decay to 0; no weight decay |
| Steps | 12,617 = 1 epoch |
| Padding | conversations sorted by length inside each step: padding waste 29% → 6% |
| Throughput | ~12.2K tokens/s (no torch.compile: every batch has a different length) |
| Validation | masked loss on the same 500 test conversations every 250 steps |
Evaluation
HellaSwag, all 10,042 validation examples, completion style as in build-nanogpt (each ending is scored by its average token loss after the context; the lowest wins; random = 25%). Same script for every row:
| Model | acc | acc_norm |
|---|---|---|
| OpenAI GPT-2 124M | 28.58% | 29.55% |
| Firstlight-124M base | 28.86% | 30.30% |
| Firstlight-124M chat | - | 29.81% |
SFT cost 0.5 points: the chat model kept most of the pretraining knowledge.
Same question, before and after SFT:
Q: Hi! Can you give me three tips for studying for an exam?
base: =/?src=?src=?src=?src=?src=?src=?src=?src=... (in the chat format: never stops)
chat: 1. **Evaluate your answers**: Make a note of your answers to the exam questions. ...
2. **Take your time**: Make a plan for studying. ...
3. **Use study materials**: Choose study materials that are relevant to your studies ...
The chat model stopped by itself on 27 of 50 test questions within 300 tokens (real answers in the data are long too: median 239 tokens).
Files
| File | Contents |
|---|---|
base/model.safetensors |
pretrained model, fp32 |
chat/model.safetensors |
SFT model, fp32 |
config.json |
model size + special token ids |
model.py |
the network, chat format and sampling |
sft_loss.png |
the SFT loss plot |
lm_head.weight is not stored: it is tied to transformer.wte.weight, and the loader re-ties it.
Credits and license
- Andrej Karpathy
- Data: FineWeb-EDU (ODC-By 1.0) and smol-smoltalk (Apache-2.0), both by Hugging Face.
- Weights and code: MIT.
