--- license: apache-2.0 language: [en] library_name: transformers pipeline_tag: text-generation tags: [udgamlabs, small-language-model, trained-from-scratch, research-release, base-model, gemma4] --- ![Laghu-Base · Udgam Labs](./banner.png) **UDGAM LABS** · Language model · 152M parameters, base # Laghu-Base Laghu-Base is the pretrained base model behind [Laghu](https://huggingface.co/udgamlabs/Laghu): 152M parameters, trained from scratch on 40.0B token positions of English web, educational, maths, code and synthetic text at a 2,048-token context. These are the exponential-moving-average (EMA) weights at the last step, 76,294. On text published in September and October 2026 it scores 0.919 bits per byte, better than SmolLM2-135M (0.941, pretrained on 2T tokens) and Falcon-H1-Tiny-90M-Base (0.979), after a $146 pretraining run. It is a text completer, not a chat model. Part of a fully open research release: code, recipe, checkpoints, evaluations and cost are all public. **Try it:** [the chat model](https://huggingface.co/udgamlabs/Laghu) · [code on GitHub](https://github.com/udgamlabs/laghu) · [udgamlabs.com/models/laghu](https://udgamlabs.com/models/laghu) ## Quick start ```python from transformers import AutoTokenizer, AutoModelForCausalLM tok = AutoTokenizer.from_pretrained("udgamlabs/Laghu-Base") model = AutoModelForCausalLM.from_pretrained("udgamlabs/Laghu-Base") inputs = tok("The water cycle begins when", return_tensors="pt") # the tokenizer adds out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95) print(tok.decode(out[0], skip_special_tokens=True)) ``` Keep prompts and outputs within 2,048 tokens: the model was trained at that length and fails at longer ones (the config allows 8,192 positions, but recall at 4K is 0%). For long inputs use [Laghu-Base-32K](https://huggingface.co/udgamlabs/Laghu-Base-32K). The raw (non-EMA) weights of the same step are in the `raw-final/` folder: `AutoModelForCausalLM.from_pretrained("udgamlabs/Laghu-Base", subfolder="raw-final")`. ## Results **Zero-shot benchmarks** (lm-eval 0.4.13, bf16, `` added, CUDA; measured by us for every model, 2026-10-19): | Model | Params | ARC-E | ARC-C | HellaSwag | PIQA | WinoGrande | 5-task avg | MMLU | GSM8K 5-shot | |---|---|---|---|---|---|---|---|---|---| | **Laghu-Base** | 152M | 60.6 | 30.1 | 40.9 | 65.7 | 52.2 | **49.9** | 25.3 | 4.2 | | GPT-X3-150M | 149M | 63.0 | 31.4 | 45.9 | 71.3 | 53.6 | 53.0 | 24.6 | 4.2 | | SmolLM2-135M | 135M | 64.5 | 30.0 | 43.0 | 68.3 | 53.0 | 51.8 | 24.5 | 2.0 | | Falcon-H1-Tiny-90M-Base | 91M | 54.3 | 28.5 | 37.7 | 64.8 | 50.8 | 47.2 | 32.4 | 2.7 | | Gargi-Base (ours) | 384M | 63.0 | 32.5 | 41.5 | 66.9 | 54.5 | 51.7 | 25.3 | 5.7 | ARC-C, HellaSwag and PIQA use length-normalised accuracy. Laghu-Base is 5th of the 19 models up to 200M parameters we measured on 19 October on the 5-task average; the full table is in the technical report. **Bits per byte on held-out text** (lower is better; one script and the same documents for every model): | Model | New text (1,695 docs: Wikipedia pages created and arXiv abstracts submitted Sep-Oct 2026) | Web (580) | Reasoning (1,179) | |---|---|---|---| | **Laghu-Base** | **0.919** | 0.858 | 0.693 | | SmolLM2-135M | 0.941 | 0.858 | 0.752 | | Falcon-H1-Tiny-90M-Base | 0.979 | 0.910 | 0.769 | | Gargi-Base (ours, 384M) | 0.914 | 0.850 | 0.686 | | SmolLM2-360M | 0.850 | 0.788 | 0.678 | ## Where it falls short - Below SmolLM2-135M (51.8) and GPT-X3-150M (53.0) on the commonsense average; near chance on MMLU (25.3). - Slightly behind our 384M Gargi-Base on all three held-out text sets (the differences are significant). - Our scaling projection before the run expected 0.900-0.912 bits per byte on new text after a 32K stretch; the measured value was 0.921, so the run came in worse than planned. - 2,048-token context only. - Not instruction-tuned: it continues text and does not follow requests. ## How it was trained - **Data:** a 52.6B-token pool from 14 public sources (DCLM-Edu and FineWeb-Edu ≈ 35%, FinePhrase ≈ 23%, Ultra-FineWeb ≈ 14.5%, FineMath ≈ 9%, SYNTH ≈ 8.5%, Wikipedia ≈ 3.4%, NVIDIA Nemotron chat and STEM ≈ 5%, permissive GitHub code ≈ 1.7%), 13-gram decontaminated against 10 evaluation sets (21,427 documents removed). Shares are the builder's own token estimates. - **Run:** 76,294 steps × 524,288 tokens = 40.0B token positions; Muon; token bags (4 tokens per position) for the first 25% of steps, so about 70B tokens were read; cool-down data mix from 80%; learning rate warm-down over the last 65% to 5% of peak; EMA from 90% (β 0.998); bf16. - **Compute and cost:** 8× A100 80GB for 13.2 hours, $146.26. About 3.6 × 10¹⁹ FLOPs. - **Architecture:** Gemma-4 text layout, 28 layers, width 640, 10 query / 2 key-value heads, 512-token sliding window with every 4th layer global, tied embeddings, our own 32,768-token tokenizer. Every setting, the data builder and the launch scripts: [GitHub](https://github.com/udgamlabs/laghu). The raw training states with optimiser at steps 76,000 and 76,294 are in [Laghu-Checkpoints](https://huggingface.co/udgamlabs/Laghu-Checkpoints); the checkpoints from steps 2,000-74,000 were deleted by a storage clean-up on 2026-10-21, our mistake. ## Intended use and limitations A base for research: fine-tuning, continued pretraining, interpretability and training-dynamics studies. It has no safety tuning and continues any text, including harmful or false text. ## Licence Apache-2.0. Copyright 2026 Udgam Labs LLP. Training-data credits: [DATA_LICENSES](https://github.com/udgamlabs/laghu/blob/main/docs/DATA_LICENSES.md). ## Citation ```bibtex @misc{udgamlabs2026laghu, title = {Laghu: a fully open 152M chat model trained from scratch}, author = {Prasad, Jitesh K}, year = {2026}, howpublished = {Udgam Labs}, url = {https://huggingface.co/udgamlabs/Laghu} } ``` --- **About Udgam Labs.** Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. [udgamlabs.com](https://udgamlabs.com)