GPT-2 from scratch
A GPT-2-style transformer implemented in PyTorch. model.safetensors contains float32 weights from training step 19,072.
Model
- 12 transformer layers, 12 attention heads, hidden size 768
- 1,024-token context window
- 50,304 vocabulary entries
- 124,475,904 unique float32 parameters
The model uses tiktoken's GPT-2 encoding (50,257 tokens) and pads its vocabulary to 50,304.
Training
Trained on 10 billion tokens from FineWeb-Edu sample-10BT using two H100 GPUs. Sources: training script, dataset loader, and hardware notes.
Results
Training loss over time
Validation loss over time
HellaSwag accuracy over time
The GPT-2 124M baselines used 100 billion training tokens; this model used 10 billion. Original plot files.
Usage
model.safetensors contains a PyTorch state dictionary, not a complete Transformers model. from_pretrained() alone cannot load it. Instantiate the project's GPT2 class with GPT2Config and load the file with safetensors.torch.load_file. No tokenizer files are included. See the source repository.
License
The model weights are independently released under ODC-By 1.0. FineWeb-Edu has its own ODC-By 1.0 license and is subject to Common Crawl's Terms of Use. The source code remains MIT-licensed.