OtakMini / README.md
syaffers's picture
Initial release: OtakMini 135M base model
b2d2e33 verified
|
Raw History Blame Contribute Delete
4.95 kB
---
license: cc-by-nc-4.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- karpathy/climbmix-400b-shuffle
- openbmb/Ultra-FineWeb
- openbmb/Ultra-FineWeb-L3
- HuggingFaceFW/fineweb-edu
- HuggingFaceFW/finephrase
- HuggingFaceTB/smollm-corpus
- nvidia/Nemotron-CC-Math-v1
- bigcode/starcoderdata
---
# OtakMini
OtakMini is a 135M-parameter English language model pretrained **from scratch** on a single consumer GPU (RTX 5050, 8 GB) in about 12 days, on 18B tokens. This is a **base model**: no instruction tuning, RLHF or safety tuning. It uses the Qwen3 architecture (grouped-query attention, QK-norm, SwiGLU, tied embeddings) so it loads with stock `transformers`.
## Model details
| | |
|---|---|
| Parameters | 135.03M (18.9M of them in the tied embedding) |
| Layers / width | 30 layers, hidden size 576, FFN 1728 |
| Attention | 9 heads, 3 KV heads, head dim 64, RoPE (theta 1e5) |
| Context length | 2048 tokens |
| Vocabulary | 32,768 (custom byte-level BPE, digits split) |
| Special tokens | `<\|endoftext\|>` (id 0: BOS/EOS/PAD), `<\|im_start\|>` (1), `<\|im_end\|>` (2, reserved for chat tuning, untrained here) |
| Precision | bf16 weights (`model.safetensors`, 270 MB) |
## Quick start
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "syaffers/OtakMini"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16)
ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids) # uses the sampling defaults in generation_config.json
print(tok.decode(out[0]))
```
`generation_config.json` ships with `temperature=0.7`, `top_p=0.9`, `repetition_penalty=1.1`.
## Evaluation
Zero-shot, full test sets, bf16, lm-evaluation-harness 0.4.13. `acc_norm` where available (HellaSwag, ARC, PIQA), otherwise `acc`.
| Task | Score | ±stderr | Chance |
|---|---|---|---|
| HellaSwag (acc_norm) | 39.9 | 0.5 | 25.0 |
| ARC-Easy (acc_norm) | 52.2 | 1.0 | 25.0 |
| ARC-Challenge (acc_norm) | 29.4 | 1.3 | 25.0 |
| PIQA (acc_norm) | 68.0 | 1.1 | 50.0 |
| Winogrande (acc) | 51.8 | 1.4 | 50.0 |
| LAMBADA (acc) | 33.9 | 0.7 | |
| LAMBADA (perplexity) | 38.2 | 1.5 | |
Validation loss on held-out web text is 2.606 (perplexity about 13.5).
## Training
- **Data:** about 19B tokens of English text, 11 sources, one pass over the pool (about 0.95 epochs). Mix for most of training, with the mix for the last 10% in brackets:
| Source | Share |
|---|---|
| ClimbMix | 30% (20%) |
| Ultra-FineWeb | 20% (15%) |
| Nemotron-CC-Math-v1 (4plus) | 10% (15%) |
| Cosmopedia v2 | 10% (12%) |
| FineWeb-Edu | 7% (10%) |
| FinePhrase (FAQ / tutorial / math / table rewrites) | 15% (20%) |
| Ultra-FineWeb-L3 (multi-style) | 5% (5%) |
| StarCoderData (Python) | 3% (3%) |
- **Decontamination:** documents with a 13-gram (word-level) overlap with HellaSwag, ARC, PIQA, Winogrande or LAMBADA were removed: 10,937 of 28.35M documents (0.04%).
- **Optimiser:** Muon (lr 0.01, weight decay 0.01) on 2D weight matrices, AdamW (lr 1.5e-3) on embeddings and norms; batch 524,288 tokens; 34,332 steps.
- **Schedule:** 2,000 warmup steps, constant lr, then a 1-sqrt cooldown over the last 10% of steps with the anneal data mix.
- **Hardware:** one RTX 5050 (8 GB) at about 17k tokens/s, 2026-10-16 to 2026-10-28.
### Loss curves
![Image of training and validation loss curves](assets/loss_curve.png)
Train loss (3-hour means) and validation loss over the run. The drop at the right is the cooldown with the anneal data mix. The hours axis is approximate.
## Intended use and limitations
Meant for research, experiments with small models, and as a starting point for fine-tuning. It is not meant for anything where correctness matters.
- It writes fluent text but **gets facts wrong** (for example, it will say Paris is the largest city in the world, or invent populations and dates), and it cannot do arithmetic reliably.
- Without a repetition penalty it **loops**. Text drifts off-topic after a sentence or two.
- English only, 2048-token context, no instruction following and no refusal behaviour. It can reproduce biases and unsafe content from web data.
- The ChatML tokens exist in the tokenizer but the model has never seen them in use. A chat template is not included.
## License and data
Released under **CC BY-NC 4.0 (non-commercial)**, because the training data includes non-commercial and agreement-restricted sources (ClimbMix is CC BY-NC 4.0; NVIDIA's Nemotron-CC-Math data agreement and the StarCoderData terms of use also apply). Check each dataset's own license before reusing the weights beyond research.
## Citation
```bibtex
@misc{otakmini2026,
title = {OtakMini: a 135M-parameter language model pretrained from scratch on a single consumer GPU},
author = {syaffers},
year = {2026},
url = {https://huggingface.co/syaffers/OtakMini}
}
```