---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
datasets:
  - HuggingFaceFW/fineweb-edu
language:
  - en
tags:
  - olmo2
  - base-model
  - gguf
  - from-scratch
  - reproducible-training
model-index:
  - name: zero-125m
    results:
      - task:
          type: text-generation
          name: Text generation
        dataset:
          name: FineWeb-Edu sample-10BT, fixed non-wrapping validation windows
          type: HuggingFaceFW/fineweb-edu
          split: validation
        metrics:
          - type: perplexity
            name: Validation perplexity
            value: 27.799
          - type: loss
            name: Validation mean loss
            value: 3.324985
      - task:
          type: text-generation
          name: Text generation
        dataset:
          name: FineWeb-Edu sample-10BT, fixed non-wrapping test windows
          type: HuggingFaceFW/fineweb-edu
          split: test
        metrics:
          - type: perplexity
            name: Test perplexity
            value: 29.970
          - type: loss
            name: Test mean loss
            value: 3.400198
---

# zero-125m

`zero-125m` is a 124,680,960-parameter OLMo2-compatible base language model
trained from scratch on 2,500,001,792 FineWeb-Edu tokens. It is the released
dense model from the
[`lm-from-zero`](https://github.com/tarpous/lm-from-zero) project, which covers
the complete path from byte-level BPE tokenization to local serving.

This repository includes the native Transformers model and tokenizer, F16 and
Q8_0 GGUF derivatives, evaluation evidence, and the full technical report.

![Local base-model continuation in Pi using the Q8_0 GGUF through llama.cpp](assets/zero-125m-serving-demo.gif)

[Technical report](lm-from-zero-technical-report.pdf) |
[Source and reproducibility](https://github.com/tarpous/lm-from-zero) |
[Capture evidence](assets/pi-demo-evidence.json)

## Intended use

The model is intended for research, education, systems experiments, conversion
tests, and local inference demonstrations. It is a pretrained base model. It
has not been aligned for assistant use and should not be treated as a reliable
chatbot, factual reference, or safety-critical system.

## Training

- Architecture: OLMo2-compatible causal decoder
- Parameters: 124,680,960
- Layers: 12
- Hidden width: 768
- Query heads: 12
- Key/value heads: 4
- Context length: 2,048 tokens
- Vocabulary: 32,000 byte-level BPE tokens
- Training tokens: 2,500,001,792
- Data: FineWeb-Edu `sample-10BT`, immutable revision
  `87f09149ef4734204d70ed1d046ddc9ca3f2b8f9`
- Optimizer: AdamW with linear warmup and cosine decay
- Numerical format: bfloat16 training with fp32 optimizer state
- Hardware: one NVIDIA RTX 4080 SUPER

The tokenizer, shard manifest, model configuration, checkpoint, evaluation,
and converted package are connected by recorded hashes. The native
Transformers export has an exact maximum fp32 logit error of 0.0 against the
project checkpoint in the recorded parity evaluation.

`model.safetensors` stores float32 tensors and is 498,738,888 bytes. That
precision is deliberate: it is what makes the exact logit-parity check above
meaningful, because the released file is bit-identical in value to the project
checkpoint rather than a re-quantized copy of it. Load with
`dtype=torch.bfloat16` for inference, or take one of the GGUF files below if a
smaller download matters more than parity.

## Evaluation

Evaluation uses fixed, non-wrapping FineWeb-Edu windows.

| Split | Mean loss | Perplexity |
|---|---:|---:|
| Validation | 3.324985 | 27.799 |
| Test | 3.400198 | 29.970 |

These measurements describe next-token likelihood under this tokenizer and
window protocol. They are not instruction-following, factuality, or safety
scores.

![Training loss against tokens consumed for the 20M and 125M dense models](figures/fineweb-training.png)

![Validation and test perplexity for the 20M and 125M dense models](figures/fineweb-evaluation.png)

The `figures/` directory also carries the 20M architecture study, the dense
ablation screening, and the local judge calibration from the wider project.

## Transformers usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tarpous/zero-125m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Once upon a time,", return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

This is a base model, so continuations may be repetitive, abrupt, or
inaccurate. A prompt formatted as a conversation does not turn it into an
instruction-following model.

## GGUF files

| File | Size | SHA-256 |
|---|---:|---|
| `zero-125m-f16.gguf` | 250,567,744 bytes | `4863f80ea315fc0713da86750f71cddc8dfbec09c0d3f8a853cee650220027ab` |
| `zero-125m-q8_0.gguf` | 133,708,864 bytes | `401866c0a19e803eea19b00e056915a37b05ebe1146430b618355071680b8692` |

The Q8_0 artifact produced 76.5 decoded tokens per second in the recorded
bounded CUDA llama.cpp smoke. Throughput is hardware and build dependent.
Conversion details and the pre-tokenizer compatibility patch are documented in
the project repository.

## Broader project evidence

The project also trained a 20M dense FineWeb model and separate TinyStories
models near 20M parameters using dense attention, Mamba-2, and masked discrete
diffusion. It includes dense ablations, supervised fine-tuning, DPO, and local
judge calibration. Those results describe the research project and are not
additional training stages of this released base checkpoint.

The full report is available in this model repository as
`lm-from-zero-technical-report.pdf` and in the GitHub release.

## Limitations

- The model is small and was trained for only 2.5 billion tokens.
- It is a base model without safety or instruction tuning.
- FineWeb-Edu may contain errors, bias, personal information, and other
  undesirable web content despite dataset-level filtering.
- The model can generate false, biased, offensive, repetitive, or incoherent
  text.
- Evaluation covers held-out language-model loss, not downstream capability or
  responsible-deployment criteria.
- The planned 125M FineWeb Mamba-2 and masked-diffusion runs were not completed.
- The planned 500-prompt cross-judge calibration was not completed.

Users are responsible for evaluating the model in their own setting and for
adding safeguards appropriate to any application.

## Citation

```bibtex
@software{tarpous2026lmfromzero,
  author = {tarpous},
  title = {lm-from-zero: A Local-First, Reproducible Language-Model Pipeline from Tokenization to Serving},
  year = {2026},
  url = {https://github.com/tarpous/lm-from-zero}
}
```

## License

The model weights, tokenizer, GGUF files, and code are released under
Apache-2.0. The technical report and its figures are released under CC BY 4.0.