--- license: cc-by-nc-4.0 language: - en library_name: transformers pipeline_tag: text-generation datasets: - karpathy/climbmix-400b-shuffle - openbmb/Ultra-FineWeb - openbmb/Ultra-FineWeb-L3 - HuggingFaceFW/fineweb-edu - HuggingFaceFW/finephrase - HuggingFaceTB/smollm-corpus - nvidia/Nemotron-CC-Math-v1 - bigcode/starcoderdata --- # OtakMini OtakMini is a 135M-parameter English language model pretrained **from scratch** on a single consumer GPU (RTX 5050, 8 GB) in about 12 days, on 18B tokens. This is a **base model**: no instruction tuning, RLHF or safety tuning. It uses the Qwen3 architecture (grouped-query attention, QK-norm, SwiGLU, tied embeddings) so it loads with stock `transformers`. ## Model details | | | |---|---| | Parameters | 135.03M (18.9M of them in the tied embedding) | | Layers / width | 30 layers, hidden size 576, FFN 1728 | | Attention | 9 heads, 3 KV heads, head dim 64, RoPE (theta 1e5) | | Context length | 2048 tokens | | Vocabulary | 32,768 (custom byte-level BPE, digits split) | | Special tokens | `<\|endoftext\|>` (id 0: BOS/EOS/PAD), `<\|im_start\|>` (1), `<\|im_end\|>` (2, reserved for chat tuning, untrained here) | | Precision | bf16 weights (`model.safetensors`, 270 MB) | ## Quick start ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer repo = "syaffers/OtakMini" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16) ids = tok("The capital of France is", return_tensors="pt") out = model.generate(**ids) # uses the sampling defaults in generation_config.json print(tok.decode(out[0])) ``` `generation_config.json` ships with `temperature=0.7`, `top_p=0.9`, `repetition_penalty=1.1`. ## Evaluation Zero-shot, full test sets, bf16, lm-evaluation-harness 0.4.13. `acc_norm` where available (HellaSwag, ARC, PIQA), otherwise `acc`. | Task | Score | ±stderr | Chance | |---|---|---|---| | HellaSwag (acc_norm) | 39.9 | 0.5 | 25.0 | | ARC-Easy (acc_norm) | 52.2 | 1.0 | 25.0 | | ARC-Challenge (acc_norm) | 29.4 | 1.3 | 25.0 | | PIQA (acc_norm) | 68.0 | 1.1 | 50.0 | | Winogrande (acc) | 51.8 | 1.4 | 50.0 | | LAMBADA (acc) | 33.9 | 0.7 | | | LAMBADA (perplexity) | 38.2 | 1.5 | | Validation loss on held-out web text is 2.606 (perplexity about 13.5). ## Training - **Data:** about 19B tokens of English text, 11 sources, one pass over the pool (about 0.95 epochs). Mix for most of training, with the mix for the last 10% in brackets: | Source | Share | |---|---| | ClimbMix | 30% (20%) | | Ultra-FineWeb | 20% (15%) | | Nemotron-CC-Math-v1 (4plus) | 10% (15%) | | Cosmopedia v2 | 10% (12%) | | FineWeb-Edu | 7% (10%) | | FinePhrase (FAQ / tutorial / math / table rewrites) | 15% (20%) | | Ultra-FineWeb-L3 (multi-style) | 5% (5%) | | StarCoderData (Python) | 3% (3%) | - **Decontamination:** documents with a 13-gram (word-level) overlap with HellaSwag, ARC, PIQA, Winogrande or LAMBADA were removed: 10,937 of 28.35M documents (0.04%). - **Optimiser:** Muon (lr 0.01, weight decay 0.01) on 2D weight matrices, AdamW (lr 1.5e-3) on embeddings and norms; batch 524,288 tokens; 34,332 steps. - **Schedule:** 2,000 warmup steps, constant lr, then a 1-sqrt cooldown over the last 10% of steps with the anneal data mix. - **Hardware:** one RTX 5050 (8 GB) at about 17k tokens/s, 2026-10-16 to 2026-10-28. ### Loss curves ![Image of training and validation loss curves](assets/loss_curve.png) Train loss (3-hour means) and validation loss over the run. The drop at the right is the cooldown with the anneal data mix. The hours axis is approximate. ## Intended use and limitations Meant for research, experiments with small models, and as a starting point for fine-tuning. It is not meant for anything where correctness matters. - It writes fluent text but **gets facts wrong** (for example, it will say Paris is the largest city in the world, or invent populations and dates), and it cannot do arithmetic reliably. - Without a repetition penalty it **loops**. Text drifts off-topic after a sentence or two. - English only, 2048-token context, no instruction following and no refusal behaviour. It can reproduce biases and unsafe content from web data. - The ChatML tokens exist in the tokenizer but the model has never seen them in use. A chat template is not included. ## License and data Released under **CC BY-NC 4.0 (non-commercial)**, because the training data includes non-commercial and agreement-restricted sources (ClimbMix is CC BY-NC 4.0; NVIDIA's Nemotron-CC-Math data agreement and the StarCoderData terms of use also apply). Check each dataset's own license before reusing the weights beyond research. ## Citation ```bibtex @misc{otakmini2026, title = {OtakMini: a 135M-parameter language model pretrained from scratch on a single consumer GPU}, author = {syaffers}, year = {2026}, url = {https://huggingface.co/syaffers/OtakMini} } ```