license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- causal-lm
- base-model
- from-scratch
- fineweb-edu
tiny-edu-114m
A 114M-parameter decoder-only language model trained from scratch (the author's second language model, after the 42M one and before Zenyx-100M-Base). It uses the GPT-2 tokenizer, which wastes much of the capacity on a 50k vocabulary that is poorly suited to its size (the embedding table alone is 38.6M of the 114M parameters); the project moved to a custom 49k tokenizer afterwards. The weights are published for completeness and for comparison; this is a base model, not instruction-tuned, and it is not recommended for use.
Previously named
tiny-edu-50m; the "50m" did not match the real size (114.1M parameters). The old name redirects here.
At a glance
| Parameters | 114,133,248 (the output head is tied to the embedding) |
| Architecture | 12 pre-norm blocks, width 768, grouped-query attention (12 query heads, 4 key/value heads, head size 64), rotary positions (theta 10,000), GELU MLP 768-3072-768, LayerNorm with bias, no linear biases |
| Vocabulary | 50,257 (GPT-2 byte-level BPE) |
| Training | 790,500 iterations, 107.9B tokens seen (training script not published; the figures come from the saved checkpoint) |
| Last logged losses | train 4.125, validation 3.817 (best validation loss logged during training: 3.612, from a checkpoint that is not kept) |
| Weights | model.safetensors (bfloat16, 228 MB) from the final checkpoint checkpoint_790500 |
| Code | modeling_tiny_edu.py (plain PyTorch, no custom transformers class) |
| License | Apache-2.0 |
The architecture was reconstructed from the checkpoint tensors (the original training code is not in this repository). The reconstruction reproduces the validation loss stored in the checkpoint: 3.82 on FineWeb-Edu text against the recorded 3.817, so the implementation is right. The exact training data mixture is not documented here; the model name and the loss on FineWeb-Edu point to an educational-web corpus.
Results
Base model, 0-shot, all test items, bfloat16 weights run in fp16. Same harness and items as the other Zenyx models (Fact1000 is a 1,000-item four-way factual test written by the author, chance 25%). Standard error is about 1.5 points on 1,000 items.
| tiny-edu-114m | Zenyx-100M-Base (step 20,300, 151M parameters, 10.6B tokens) | |
|---|---|---|
| Fact1000 | 48.5% | 73.5% |
| Fact1000 without the arithmetic items | 54.0% | 70.7% |
| SciQ (acc / length-normalised) | 61.2 / 55.1 | 79.5 / 74.8 |
| ARC-Easy | 40.1 / 36.7 | 47.3 / 44.0 |
| ARC-Challenge | 17.3 / 22.5 | 22.1 / 25.3 |
| Pile bits per byte, 22 domains (lower is better) | 1.721 | 1.071 |
| Pile, mean of domains | 1.863 | 1.223 |
Bits per byte does not depend on the tokenizer, so the Pile rows are a fair comparison. Zenyx-100M-Base scores much higher with a tenth of the training tokens. The likely causes are the tokenizer and the data mixture (not tested separately).
Usage
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoTokenizer
from modeling_tiny_edu import TinyEdu, generate # download modeling_tiny_edu.py from this repository
model = TinyEdu(); model.load_state_dict(load_file(hf_hub_download("Arko007/tiny-edu-114m", "model.safetensors")), strict=False) # strict=False: the output head is tied
model = model.float().eval()
tok = AutoTokenizer.from_pretrained("gpt2")
print(generate(model, tok, "The capital of France is", max_new=30))
Sampled continuations (temperature 0.8, top-k 50, one sample each; generated with the code above):
The capital of France is Germany, and the largest city in southern France is Germany. The French state is the biggest city in France, with the capital of France a third. With its capital, the city is home to
Photosynthesis is an important role in the production of biomass. In particular, it has been suggested that the uptake of biomass from the soil is the key to improving plant productivity. At the same time, biomass contains essential
In 1969, the first humans to be fully independent were humans. “If we were to be independent, we would have to be independent” (Battles, p. 39). In 1974, the Human Rights
To make a good cup of tea, you should have a cup of coffee in the middle of the cup, and a cup of coffee that you can use for a week. A cup of coffee can be used to make a cup of tea or
Limitations
- Base model: it continues text, makes up facts, repeats itself in long outputs, and does not follow instructions.
- English only, GPT-2 tokenizer (poor on code, digits and non-English text), not evaluated for bias or harmful content.
- Numbers on this card come from one checkpoint and the author's own harness; Fact1000 is written by the author.
- The original training code and data mixture are not published, and 633 intermediate checkpoints (iterations 473,000 to 790,000) were removed from this repository to keep it small; only the final weights remain.