Card and NOTICE (private staging)

README.md

@@ -0,0 +1,109 @@
+---
+license: apache-2.0
+language: [en]
+tags: [udgamlabs, small-language-model, trained-from-scratch, research-release, checkpoints, training-dynamics]
+---
+
+![Laghu-Checkpoints · Udgam Labs](./banner.png)
+
+**UDGAM LABS** · Training checkpoints · 152M parameters, every surviving state
+
+# Laghu-Checkpoints
+
+The raw training states behind [Laghu](https://huggingface.co/udgamlabs/Laghu), a 152M-parameter chat model trained from
+scratch for $146 of pretraining compute: model weights, EMA weights and optimiser states from the end of pretraining
+and of the 32K stretch, all three supervised fine-tuning runs (1, 2 and 3 passes), the reinforcement-learning run that
+failed its gate, and the run logs. Together with the code on GitHub, they let you resume training, re-run the
+fine-tuning, or study how the model changed between stages.
+
+**Try it:** [the chat model](https://huggingface.co/udgamlabs/Laghu) · [code on GitHub](https://github.com/udgamlabs/laghu) · [udgamlabs.com/models/laghu](https://udgamlabs.com/models/laghu)
+
+## Quick start
+
+```python
+import torch
+from huggingface_hub import hf_hub_download
+
+p = hf_hub_download("udgamlabs/Laghu-Checkpoints", "pretrain/step_076294/model_ema_076294.pt")
+state = torch.load(p, map_location="cpu", weights_only=True)   # a state dict for nanochat's GPT model
+```
+
+These are nanochat files (`torch.save`), not transformers checkpoints. Load them with the training code in
+[udgamlabs/laghu](https://github.com/udgamlabs/laghu) (`nanochat/`), or convert one to the transformers format with
+`python -m scripts.export_hf --checkpoint-dir <folder> --weights final --out <dir>`. Ready-to-use transformers weights
+are in [Laghu](https://huggingface.co/udgamlabs/Laghu), [Laghu-Base](https://huggingface.co/udgamlabs/Laghu-Base) and
+[Laghu-Base-32K](https://huggingface.co/udgamlabs/Laghu-Base-32K), and in the `hf/` folders below.
+
+## What is here
+
+| Folder | Stage | Files |
+|---|---|---|
+| `pretrain/step_076000/` | pretraining, 99.6% | `model_076000.pt` (weights), `ema_076000.pt` (EMA), `optim_076000_rank{0-7}.pt` (one optimiser shard per GPU of the 8-GPU run), `meta_076000.json` |
+| `pretrain/step_076294/` | end of pretraining | as above, plus `model_ema_076294.pt` (the EMA weights as a state dict; this is Laghu-Base) |
+| `pretrain/logs/`, `pretrain/summary.txt` | | the main run's log (every step's loss, validation every 2,000 steps) |
+| `ctx32k/step_001750/`, `ctx32k/step_001900/` | 32K stretch | `model_*.pt`, `optim_*_rank0.pt`, `meta_*.json`; step 1,900 is Laghu-Base-32K and the start of SFT |
+| `ctx32k/logs/`, `ctx32k/eval/` | | training log; recall and short-text checks |
+| `sft/1-pass/`, `sft/2-passes/`, `sft/3-passes/` | supervised fine-tuning | `ckpt/` (weights and optimiser at the halfway and final steps), `hf/` (transformers export of the final step), `logs/`; 2 passes is the released Laghu |
+| `rl-r1-failed/` | reinforcement learning (GRPO + distillation from Gargi), 40 steps | `hf/` export, `eval/`, `logs/`; it failed its pre-registered gate (dev GSM8K 8.6% → 8.2%, bar 11.6%) and is published for study only |
+
+File formats:
+- `model_*.pt` and `model_ema_*.pt`: state dicts (the EMA ones in fp32), loadable with `torch.load(..., weights_only=True)`.
+- `ema_*.pt`: the EMA as a list of fp32 tensors in `model.parameters()` order, used to resume the EMA.
+- `optim_*.pt`: optimiser state dicts (Muon for matrices, Adam for embeddings, output and scalars), loadable with
+  `weights_only=True`.
+- `meta_*.json`: model config, run settings, data-loader position and validation scores at that step. Measure a
+  checkpoint yourself rather than reading scores from its meta file.
+
+## Results
+
+Measured on the exported weights (details on the model cards and in the
+[technical report](https://udgamlabs.com/research/laghu)):
+
+| Checkpoint | Bits per byte on new text (lower is better) | Notes |
+|---|---|---|
+| pretraining step 76,294, EMA (Laghu-Base) | 0.919 | SmolLM2-135M 0.941, Falcon-H1-Tiny-90M-Base 0.979 |
+| 32K step 1,900 (Laghu-Base-32K) | 0.921 | hidden-code recall 96-100% up to 16K tokens |
+| SFT 1 / 2 / 3 passes | held-out SFT loss 1.724 / 1.784 / 2.036 | 2 passes was picked by a generation-based rule (60.0% judge win rate over 1 pass) |
+
+## Where it falls short
+
+- **Most of the pretraining trajectory is missing.** Checkpoints were saved every 2,000 steps, but the uploader kept
+  only the newest two on the main branch, and a storage clean-up on 2026-10-21 deleted the older versions (steps
+  2,000-74,000) and about six earlier 32K checkpoints. That was our mistake. Only the late states here survive, so this
+  repository cannot show how Laghu learned over the whole run.
+- The 32K stretch states have one optimiser shard (one GPU); pretraining states have eight. Resuming on a different GPU
+  count needs re-sharding.
+- The RL run is a failed experiment, not an improvement.
+
+## How it was trained
+
+Pretraining: 76,294 steps × 524,288 tokens (40.0B token positions) at 2,048 context on 8× A100 80GB, 13.2 hours, $146.26;
+Muon; token bags for the first 25% of steps; cool-down data from 80%; EMA from 90% (β 0.998). 32K stretch: 1,900 steps at
+32,768 context on one RTX PRO 6000. SFT: 41,467 conversations, sequence 16,384, batch 262,144 tokens. Every setting and
+script: [GitHub](https://github.com/udgamlabs/laghu).
+
+## Intended use and limitations
+
+Research on training dynamics, continued pretraining, fine-tuning and reproduction. The weights carry no safety tuning
+beyond light SFT data.
+
+## Licence
+
+Apache-2.0. Copyright 2026 Udgam Labs LLP. Training-data credits:
+[DATA_LICENSES](https://github.com/udgamlabs/laghu/blob/main/docs/DATA_LICENSES.md).
+
+## Citation
+
+```bibtex
+@misc{udgamlabs2026laghu,
+  title  = {Laghu: a fully open 152M chat model trained from scratch},
+  author = {Prasad, Jitesh K},
+  year   = {2026},
+  howpublished = {Udgam Labs},
+  url    = {https://huggingface.co/udgamlabs/Laghu}
+}
+```
+
+---
+
+**About Udgam Labs.** Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. [udgamlabs.com](https://udgamlabs.com)