firstlight-124m / README.md
adarsh-007's picture
Card: link the GitHub code
8799056 verified
|
Raw History Blame Contribute Delete
8.11 kB
metadata
license: mit
library_name: pytorch
pipeline_tag: text-generation
language:
  - en
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceTB/smol-smoltalk
tags:
  - gpt2
  - from-scratch
  - pretraining
  - sft
  - chat
  - nanochat

Firstlight-124M

A 124M-parameter language model pretrained from scratch on 9.83B tokens of FineWeb-EDU, then turned into a chat assistant with supervised fine-tuning (SFT) on smol-smoltalk. GPT-2 architecture, written in plain PyTorch.

Two models in this repo:

  • base/: the pretrained model. It continues text.
  • chat/: the base model after SFT. It answers a question and stops.

Try it: Firstlight Chat (the full model, hosted on Modal; the first visit after a quiet spell takes ~30 s to start). The "Base vs Chat" tab sends the same prompt to both models.

Code: github.com/adarsh070502/firstlight (pretraining, SFT, evals, release and demo).

Model HellaSwag (acc_norm)
OpenAI GPT-2 124M 29.55%
Firstlight-124M base 30.30%
Firstlight-124M chat 29.81%

Model details

Developed by Adarsh (adarsh-007)
Type decoder-only transformer (GPT-2): 12 layers, 12 heads, 768 dim, 1024-token context
Parameters 124,475,904
Tokenizer GPT-2 BPE (tiktoken) + 4 chat tokens
Language English
License MIT

Uses

  • Direct use: learning and demos: how pretraining and SFT change a model's behaviour, small-model experiments, a starting point for preference tuning (DPO) or LoRA experiments.
  • Out of scope: anything that needs correct facts or safe output. It has no preference tuning and no safety training.

How to use

The network, chat format and sampling (with a KV cache) are in model.py in this repo (needs torch, tiktoken, safetensors, huggingface_hub):

import shutil
from huggingface_hub import hf_hub_download
shutil.copy(hf_hub_download("adarsh-007/firstlight-124m", "model.py"), "model.py")

from model import load_model, render_chat, stream, enc

chat = load_model("chat")
ids = render_chat([{"role": "user", "content": "What is photosynthesis?"}])
reply = ""
for reply in stream(chat, ids, max_new_tokens=300, temperature=0.7, top_k=50):
    pass
print(reply)

base = load_model("base")                 # continues plain text
text = ""
for text in stream(base, enc.encode("The best way to learn a language is"), chat=False):
    pass
print(text)

Chat format:

<|endoftext|> <|user_start|> What is 2+2? <|user_end|> <|assistant_start|> It is 4. <|assistant_end|>

The four chat tokens use ids 50257–50260, unused slots of the vocabulary (padded from 50,257 to 50,304).

Limitations

At 124M parameters the chat model gets the format right (direct answers, numbered lists, bold headers, stopping by itself) but not the content: it states wrong facts confidently ("Paris ... about 2,500 miles south of the City of Light"), miscounts (asked for 3 tips, gives 5 or 8) and repeats itself in long answers. SFT teaches behaviour, not knowledge.

Multi-turn chat works ("Can you make it shorter?" rewrites its previous answer), but small talk is weak: a bare "thanks" often gets the topic repeated instead of "you're welcome". smol-smoltalk is almost all task conversations, so the model rarely saw that kind of turn.

Both models see at most 1,024 tokens (about 750 words). In long chats the oldest turns are forgotten.

Training

Pretraining (base model)

  • Data: FineWeb-EDU sample-10BT, 9.83B tokens (1 epoch).
  • Recipe: 18,751 steps × 0.5M tokens, AdamW, LR 6e-4 with warmup and cosine decay, bf16.
  • Result: val loss 3.0436; HellaSwag 30.30% vs 29.55% for OpenAI's GPT-2 124M.
Hyperparameters
Steps 18,751 × 524,288 tokens (2^19)
Micro-batch 4 × 1024 tokens, 128 gradient-accumulation steps per optimizer step
Optimizer AdamW, betas (0.9, 0.95), weight decay 0.1 on 2-D weights, grad clip 1.0
Learning rate linear warmup 715 steps to 6e-4, cosine decay to 6e-5
Precision bf16 autocast, TF32 matmuls, torch.compile, flash attention
Throughput ~16.4K tokens/s, ~7 days
Validation first 100M-token shard held out; best val loss at step 18,250
Checkpoints every 100 steps (atomic write); resumed twice on the exact next tokens

SFT (chat model)

  • Data: smol-smoltalk, 403,748 conversations (253M tokens), a dataset made for models under 1B.
  • Loss: the same next-token loss as pretraining, only on the assistant's tokens and <|assistant_end|> (so it learns to stop); user tokens and padding are ignored.
  • Recipe: 1 epoch, 12,617 steps × 32 conversations, LR 5e-5, fresh AdamW, 5.8 hours.
  • Result: val loss on assistant tokens 2.478 → 1.4495, with no overfitting.

SFT loss

Data filtering
Train Test
raw conversations 460,341 24,229
dropped: Python coding (self-oss-instruct); FineWeb-EDU has almost no code 48,071 2,590
dropped: the first answer alone is longer than 1,024 tokens 8,522 448
cut after the last complete assistant turn that fits in 1,024 tokens 178,102 9,429
kept 403,748 21,191
tokens 253M (81% trained on) 13M
Hyperparameters
Start base model weights, a fresh AdamW
Batch 32 conversations per step (micro-batch 4 × 8 accumulation), one conversation per row, right-padded
Learning rate 5e-5, warmup 50 steps, linear decay to 0; no weight decay
Steps 12,617 = 1 epoch
Padding conversations sorted by length inside each step: padding waste 29% → 6%
Throughput ~12.2K tokens/s (no torch.compile: every batch has a different length)
Validation masked loss on the same 500 test conversations every 250 steps

Evaluation

HellaSwag, all 10,042 validation examples, completion style as in build-nanogpt (each ending is scored by its average token loss after the context; the lowest wins; random = 25%). Same script for every row:

Model acc acc_norm
OpenAI GPT-2 124M 28.58% 29.55%
Firstlight-124M base 28.86% 30.30%
Firstlight-124M chat - 29.81%

SFT cost 0.5 points: the chat model kept most of the pretraining knowledge.

Same question, before and after SFT:

Q: Hi! Can you give me three tips for studying for an exam?

base: =/?src=?src=?src=?src=?src=?src=?src=?src=...        (in the chat format: never stops)
chat: 1. **Evaluate your answers**: Make a note of your answers to the exam questions. ...
      2. **Take your time**: Make a plan for studying. ...
      3. **Use study materials**: Choose study materials that are relevant to your studies ...

The chat model stopped by itself on 27 of 50 test questions within 300 tokens (real answers in the data are long too: median 239 tokens).

Files

File Contents
base/model.safetensors pretrained model, fp32
chat/model.safetensors SFT model, fp32
config.json model size + special token ids
model.py the network, chat format and sampling
sft_loss.png the SFT loss plot

lm_head.weight is not stored: it is tied to transformer.wte.weight, and the loader re-ties it.

Credits and license

  • Andrej Karpathy
  • Data: FineWeb-EDU (ODC-By 1.0) and smol-smoltalk (Apache-2.0), both by Hugging Face.
  • Weights and code: MIT.