| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| language: |
| - en |
| tags: |
| - olmo2 |
| - base-model |
| - gguf |
| - from-scratch |
| - reproducible-training |
| model-index: |
| - name: zero-125m |
| results: |
| - task: |
| type: text-generation |
| name: Text generation |
| dataset: |
| name: FineWeb-Edu sample-10BT, fixed non-wrapping validation windows |
| type: HuggingFaceFW/fineweb-edu |
| split: validation |
| metrics: |
| - type: perplexity |
| name: Validation perplexity |
| value: 27.799 |
| - type: loss |
| name: Validation mean loss |
| value: 3.324985 |
| - task: |
| type: text-generation |
| name: Text generation |
| dataset: |
| name: FineWeb-Edu sample-10BT, fixed non-wrapping test windows |
| type: HuggingFaceFW/fineweb-edu |
| split: test |
| metrics: |
| - type: perplexity |
| name: Test perplexity |
| value: 29.970 |
| - type: loss |
| name: Test mean loss |
| value: 3.400198 |
| --- |
| |
| # zero-125m |
|
|
| `zero-125m` is a 124,680,960-parameter OLMo2-compatible base language model |
| trained from scratch on 2,500,001,792 FineWeb-Edu tokens. It is the released |
| dense model from the |
| [`lm-from-zero`](https://github.com/tarpous/lm-from-zero) project, which covers |
| the complete path from byte-level BPE tokenization to local serving. |
|
|
| This repository includes the native Transformers model and tokenizer, F16 and |
| Q8_0 GGUF derivatives, evaluation evidence, and the full technical report. |
| |
|  |
| |
| [Technical report](lm-from-zero-technical-report.pdf) | |
| [Source and reproducibility](https://github.com/tarpous/lm-from-zero) | |
| [Capture evidence](assets/pi-demo-evidence.json) |
| |
| ## Intended use |
| |
| The model is intended for research, education, systems experiments, conversion |
| tests, and local inference demonstrations. It is a pretrained base model. It |
| has not been aligned for assistant use and should not be treated as a reliable |
| chatbot, factual reference, or safety-critical system. |
| |
| ## Training |
| |
| - Architecture: OLMo2-compatible causal decoder |
| - Parameters: 124,680,960 |
| - Layers: 12 |
| - Hidden width: 768 |
| - Query heads: 12 |
| - Key/value heads: 4 |
| - Context length: 2,048 tokens |
| - Vocabulary: 32,000 byte-level BPE tokens |
| - Training tokens: 2,500,001,792 |
| - Data: FineWeb-Edu `sample-10BT`, immutable revision |
| `87f09149ef4734204d70ed1d046ddc9ca3f2b8f9` |
| - Optimizer: AdamW with linear warmup and cosine decay |
| - Numerical format: bfloat16 training with fp32 optimizer state |
| - Hardware: one NVIDIA RTX 4080 SUPER |
| |
| The tokenizer, shard manifest, model configuration, checkpoint, evaluation, |
| and converted package are connected by recorded hashes. The native |
| Transformers export has an exact maximum fp32 logit error of 0.0 against the |
| project checkpoint in the recorded parity evaluation. |
| |
| `model.safetensors` stores float32 tensors and is 498,738,888 bytes. That |
| precision is deliberate: it is what makes the exact logit-parity check above |
| meaningful, because the released file is bit-identical in value to the project |
| checkpoint rather than a re-quantized copy of it. Load with |
| `dtype=torch.bfloat16` for inference, or take one of the GGUF files below if a |
| smaller download matters more than parity. |
| |
| ## Evaluation |
| |
| Evaluation uses fixed, non-wrapping FineWeb-Edu windows. |
| |
| | Split | Mean loss | Perplexity | |
| |---|---:|---:| |
| | Validation | 3.324985 | 27.799 | |
| | Test | 3.400198 | 29.970 | |
| |
| These measurements describe next-token likelihood under this tokenizer and |
| window protocol. They are not instruction-following, factuality, or safety |
| scores. |
| |
|  |
| |
|  |
| |
| The `figures/` directory also carries the 20M architecture study, the dense |
| ablation screening, and the local judge calibration from the wider project. |
| |
| ## Transformers usage |
| |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "tarpous/zero-125m" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| |
| inputs = tokenizer("Once upon a time,", return_tensors="pt").to(model.device) |
| with torch.inference_mode(): |
| output = model.generate( |
| **inputs, |
| max_new_tokens=64, |
| do_sample=False, |
| ) |
| |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
| |
| This is a base model, so continuations may be repetitive, abrupt, or |
| inaccurate. A prompt formatted as a conversation does not turn it into an |
| instruction-following model. |
| |
| ## GGUF files |
| |
| | File | Size | SHA-256 | |
| |---|---:|---| |
| | `zero-125m-f16.gguf` | 250,567,744 bytes | `4863f80ea315fc0713da86750f71cddc8dfbec09c0d3f8a853cee650220027ab` | |
| | `zero-125m-q8_0.gguf` | 133,708,864 bytes | `401866c0a19e803eea19b00e056915a37b05ebe1146430b618355071680b8692` | |
| |
| The Q8_0 artifact produced 76.5 decoded tokens per second in the recorded |
| bounded CUDA llama.cpp smoke. Throughput is hardware and build dependent. |
| Conversion details and the pre-tokenizer compatibility patch are documented in |
| the project repository. |
| |
| ## Broader project evidence |
| |
| The project also trained a 20M dense FineWeb model and separate TinyStories |
| models near 20M parameters using dense attention, Mamba-2, and masked discrete |
| diffusion. It includes dense ablations, supervised fine-tuning, DPO, and local |
| judge calibration. Those results describe the research project and are not |
| additional training stages of this released base checkpoint. |
| |
| The full report is available in this model repository as |
| `lm-from-zero-technical-report.pdf` and in the GitHub release. |
| |
| ## Limitations |
| |
| - The model is small and was trained for only 2.5 billion tokens. |
| - It is a base model without safety or instruction tuning. |
| - FineWeb-Edu may contain errors, bias, personal information, and other |
| undesirable web content despite dataset-level filtering. |
| - The model can generate false, biased, offensive, repetitive, or incoherent |
| text. |
| - Evaluation covers held-out language-model loss, not downstream capability or |
| responsible-deployment criteria. |
| - The planned 125M FineWeb Mamba-2 and masked-diffusion runs were not completed. |
| - The planned 500-prompt cross-judge calibration was not completed. |
| |
| Users are responsible for evaluating the model in their own setting and for |
| adding safeguards appropriate to any application. |
| |
| ## Citation |
| |
| ```bibtex |
| @software{tarpous2026lmfromzero, |
| author = {tarpous}, |
| title = {lm-from-zero: A Local-First, Reproducible Language-Model Pipeline from Tokenization to Serving}, |
| year = {2026}, |
| url = {https://github.com/tarpous/lm-from-zero} |
| } |
| ``` |
| |
| ## License |
| |
| The model weights, tokenizer, GGUF files, and code are released under |
| Apache-2.0. The technical report and its figures are released under CC BY 4.0. |
| |
