cloudkites/gpt2-124m-fineweb-edu

GPT-2 124M, trained from scratch on FineWeb-Edu (10B tokens)

Open weights. A GPT-2 124M model pretrained from random initialisation, released so the result in our paper can be checked against the artefact that produced it.

Parameters	124,475,904 (148 tensors)
Architecture	GPT-2 124M: 12 layers, 12 heads, 768 hidden, 1024 context
Tokenizer	GPT-2 byte-level BPE (unchanged)
Vocabulary	50,304 rows, of which the first 50,257 are scored
Training data	FineWeb-Edu, the ten-billion-token sample
Tokens seen	9.91 B (18,865 optimizer steps x 525,312 tokens)
Precision	bfloat16 activations over float32 master weights
Released weights	float32
Held-out cross-entropy	3.2598
HellaSwag (length-normalised)	0.3036 (0.2895 unnormalised, 10,042 items)

What these numbers mean

The reference point is Karpathy's llm.c, which reports 3.29 held-out loss and 0.299 HellaSwag for the same model and corpus. This run lands at 3.2598 and 0.3036.

We read that as reproducing the reference, not as beating it. Our optimizer step is 525,312 tokens against the reference's 524,288 -- 0.195% more -- because a three-GPU arrangement cannot divide 524,288 into a whole number of sequences. Across an equal number of steps this run therefore saw slightly more data, which is the most likely explanation for a margin of that size.

eval_curve.json carries all 76 held-out evaluations, so the trajectory can be inspected rather than taken on trust.

Tensor names

Names follow llm.c's flat convention, not HuggingFace's:

wte                     [50304, 768]     token embedding (tied to the LM head)
wpe                     [1024, 768]      learned position embedding
h.<i>.ln_1.{weight,bias}
h.<i>.attn.c_attn.{weight,bias}          fused q,k,v
h.<i>.attn.c_proj.{weight,bias}
h.<i>.ln_2.{weight,bias}
h.<i>.mlp.c_fc.{weight,bias}
h.<i>.mlp.c_proj.{weight,bias}
ln_f.{weight,bias}

To load with HuggingFace transformers, prefix with transformer. and rename wte/wpe to transformer.wte.weight/transformer.wpe.weight. Reading the raw file needs nothing but safetensors:

from safetensors.torch import load_file
sd = load_file("model.safetensors")
print(len(sd), sum(t.numel() for t in sd.values()))   # 148 124475904

The vocabulary is padded from 50,257 to 50,304 so the head's matrix multiply is aligned. The padding rows are never scored and receive no gradient; if you compute perplexity over all 50,304 rows you will get a different number from ours.

Training framework

Trained with numbat, a machine-learning stack written in Zig with no third-party runtime dependencies. This repository contains weights, a metric file and this card only -- no framework source, and none is required to use the weights.

Intended use and limits

A 124M-parameter base language model trained on 9.91 B tokens of web text. It is a research artefact: useful for studying pretraining reproducibility, for distillation and probing experiments, and as a small base to fine-tune. It is not instruction-tuned, not aligned, and not filtered for safety. It will produce fluent text that is frequently wrong, and it reflects the biases of its corpus. Do not deploy it to end users without your own evaluation and safeguards.

Licence and attribution

Weights released under the Apache License 2.0.

• Architecture follows OpenAI's GPT-2 (MIT-licensed reference implementation).
• Trained on FineWeb-Edu (ODC-By 1.0), which governs the data; please honour its attribution terms if you redistribute derived datasets.
• The reproduction target is llm.c (MIT).

Citation

@misc{cloudkites_gpt2_124m_fineweb_edu,
  title  = {GPT-2 124M trained from scratch on FineWeb-Edu},
  author = {Tran, Thang and Dang, Lan},
  year   = {2026},
  note   = {CloudKites AI Lab. Open weights, Apache-2.0.}
}