README.md

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - tiny-lm
  - qwen3
  - value-residual
  - muon
  - data-mixture
  - ablation
datasets:
  - HuggingFaceFW/fineweb-edu
  - mlfoundations/dclm-baseline-1.0
  - HuggingFaceTB/finemath
---

# JugnuLM-110M-R3 (value residuals + Muon + data blend)

**What "R3" means.** This is **rung 3 of the JugnuLM phase-2 ablation ladder.** We take
rung **R2** — [JugnuLM-110M-R2](https://huggingface.co/altslate/JugnuLM-110M-R2) (value
residuals + Muon) — and change **one** thing: the training **data**. Instead of
single-source FineWeb-Edu, R3 trains on a **blend** (per-sequence weighted mix), holding
architecture, optimizer, schedule, tokens, batch, and seed fixed.

## What changed: the data blend

| source | weight | role |
|---|---|---|
| FineWeb-Edu (sample-10BT) | 55% | language + general knowledge |
| DCLM-baseline | 35% | diversity → broader distribution |
| FineMath-4plus | 10% | light math reasoning |

Sequences are drawn per-step from a weighted-random source (a shuffled blend, not a staged
curriculum). Validation stayed a **fixed FineWeb-Edu held-out set** for comparable curves.

## Result — an honest negative for ARC

| 110M rung | BLiMP ↑ | ARC-Easy ↑ | WikiText-2 byte-ppl ↓ |
|---|---|---|---|
| R0 — baseline | 81.25 | 52.48 | 1.95 |
| R1 — + value residuals | 81.10 | 54.67 | 1.94 |
| R2 — + Muon | 80.78 | **56.10** | 1.932 |
| **R3 — + data blend** | **81.79** | 53.62 | **1.9092** |

**R3 has the best BLiMP and best perplexity of any rung — but ARC-Easy *dropped* −2.48 vs
R2 (56.10 → 53.62).** Since ARC-Easy is our binding constraint, **R3 is *not* kept in the
ladder** (it does not beat R2 on the metric that gates leaderboard rank; its EFF is marginally
below R2's). We publish it anyway, as the ablation result and as our strongest grammar/perplexity
checkpoint.

### Why the blend helped BLiMP/perplexity but hurt ARC
FineWeb-Edu's **educational-quality filtering** is closely aligned with ARC-Easy's
grade-school science questions. Diluting it 100% → 55% with general web text (DCLM) and math
(FineMath) improved broad language modeling (grammar, perplexity) but **removed ARC-relevant
educational signal** — DCLM is diverse but less science-dense, and FineMath targets math
reasoning, not ARC science. The lesson: at this scale, **for ARC-Easy the educational
distribution matters more than raw diversity.** Evaluated with EleutherAI `lm-evaluation-harness`.

## ⚠️ Loading — requires `trust_remote_code`

Value residuals are a **custom attention pathway**, so this model must be loaded with the
included modeling file. Loading it as a stock Qwen3 **silently drops the residual** and
understates quality.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R3")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R3", trust_remote_code=True)
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
```

## Architecture & training

Identical architecture/optimizer to R2 (value residuals + Muon): Qwen3-arch, **23 layers ×
576**, GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152),
z-loss 1e-4. Trained from scratch on ~8.4B tokens of the 55/35/10 blend above, Muon (peak
2e-2) + AdamW (1.5e-3) on a shared cosine, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO
4500 Blackwell. Only the data differs from R2.

## Intended use & limitations

Research **base** model (not instruction-tuned). Best-in-family English fluency/grammar and
lowest perplexity, but weaker ARC-Easy reasoning than R1/R2. Occasional repetition,
English-only. Not for production.

## Links

- Previous rung (best ARC): [altslate/JugnuLM-110M-R2](https://huggingface.co/altslate/JugnuLM-110M-R2)
- Baseline: [altslate/JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
- Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu

## Paper

This model is described in [*Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M*](https://arxiv.org/abs/2609.14715)
(arXiv:2609.14715 · [HF Papers](https://huggingface.co/papers/2609.14715)).

```bibtex
@misc{rajput2026jugnu,
  title         = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
  author        = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
  year          = {2026},
  eprint        = {2609.14715},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.14715}
}
```