---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - tiny-lm
  - qwen3
  - value-residual
  - resformer
  - ablation
datasets:
  - HuggingFaceFW/fineweb-edu
---

# JugnuLM-110M-R1 (value residuals)

**What "R1" means.** This is **rung 1 of the JugnuLM phase-2 ablation ladder.** We take
our honest 110M baseline — [JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
(rung **R0**) — and add **one** architectural change, **value residuals** (ResFormer),
holding *everything else fixed* (same data, tokens, schedule, batch, seed policy). The
ladder keeps a lever only if it beats the previous rung. **R1 does, so it's kept.**

## What value residuals are

Each attention layer's value gains a **learned-gated residual from the first layer's
value**:

> `v_i = v_proj_i(x) + λ_i · v₀`   (v₀ = layer-0 value; λ_i learned per layer)

This gives deep layers direct access to early-layer content (Zhou et al., *Value Residual
Learning*, ACL 2025 — [arXiv:2410.17897](https://arxiv.org/abs/2410.17897)). The 22 learned
gates converge to non-trivial values (range −2.5 to +3.2), i.e. the path is genuinely used.

## Result vs the R0 baseline

| 110M variant | BLiMP ↑ | ARC-Easy ↑ | WikiText-2 byte-ppl ↓ |
|---|---|---|---|
| R0 — baseline | 81.25 | 52.48 | 1.95 |
| **R1 — + value residuals** | 81.10 | **54.67** | **1.94** |

**Value residuals lift ARC-Easy by +2.2 at a BLiMP tie (SE ≈ 0.14) and a slightly lower
perplexity** — recovering part of the reasoning gap to the 125–143M leaderboard models,
at negligible parameter cost. Evaluated with EleutherAI `lm-evaluation-harness`.

## ⚠️ Loading — requires `trust_remote_code`

Value residuals are a **custom attention pathway**, so this model must be loaded with the
included modeling file. Loading it as a stock Qwen3 **silently drops the residual** and
understates quality.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R1")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R1", trust_remote_code=True)
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
```

## Architecture & training

Identical to R0 except the value-residual pathway: Qwen3-arch, **23 layers × 576**,
GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152),
z-loss 1e-4. Trained from scratch on ~8B tokens of FineWeb-Edu (sample-10BT), cosine LR
1.5e-3→1.5e-4, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO 4500 Blackwell.

## Intended use & limitations

Research **base** model (not instruction-tuned). Fluent English continuation and
reasonable recall for its size; limited multi-step reasoning, occasional repetition,
English-only. Not for production.

## Links

- Baseline: [altslate/JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
- Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu

## Paper

This model is described in [*Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M*](https://arxiv.org/abs/2609.14715)
(arXiv:2609.14715 · [HF Papers](https://huggingface.co/papers/2609.14715)).

```bibtex
@misc{rajput2026jugnu,
  title         = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
  author        = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
  year          = {2026},
  eprint        = {2609.14715},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.14715}
}
```