Instructions to use altslate/JugnuLM-110M-R1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use altslate/JugnuLM-110M-R1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="altslate/JugnuLM-110M-R1", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R1", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use altslate/JugnuLM-110M-R1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "altslate/JugnuLM-110M-R1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "altslate/JugnuLM-110M-R1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/altslate/JugnuLM-110M-R1

SGLang

How to use altslate/JugnuLM-110M-R1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "altslate/JugnuLM-110M-R1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "altslate/JugnuLM-110M-R1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "altslate/JugnuLM-110M-R1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "altslate/JugnuLM-110M-R1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use altslate/JugnuLM-110M-R1 with Docker Model Runner:
```
docker model run hf.co/altslate/JugnuLM-110M-R1
```

JugnuLM-110M-R1 / README.md

rajpdus

Model card: cite arXiv:2609.14715

1d488c3 verified 22 days ago

preview code

Raw History Blame Contribute Delete

3.73 kB

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - tiny-lm
  - qwen3
  - value-residual
  - resformer
  - ablation
datasets:
  - HuggingFaceFW/fineweb-edu
---
# JugnuLM-110M-R1 (value residuals)

**What "R1" means.** This is **rung 1 of the JugnuLM phase-2 ablation ladder.** We take
our honest 110M baseline — [JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
(rung **R0**) — and add **one** architectural change, **value residuals** (ResFormer),
holding *everything else fixed* (same data, tokens, schedule, batch, seed policy). The
ladder keeps a lever only if it beats the previous rung. **R1 does, so it's kept.**

## What value residuals are

Each attention layer's value gains a **learned-gated residual from the first layer's
value**:

> `v_i = v_proj_i(x) + λ_i · v₀`   (v₀ = layer-0 value; λ_i learned per layer)
This gives deep layers direct access to early-layer content (Zhou et al., *Value Residual
Learning*, ACL 2025 — [arXiv:2410.17897](https://arxiv.org/abs/2410.17897)). The 22 learned
gates converge to non-trivial values (range −2.5 to +3.2), i.e. the path is genuinely used.
## Result vs the R0 baseline
| 110M variant | BLiMP ↑ | ARC-Easy ↑ | WikiText-2 byte-ppl ↓ |
|---|---|---|---|
| R0 — baseline | 81.25 | 52.48 | 1.95 |
| **R1 — + value residuals** | 81.10 | **54.67** | **1.94** |
**Value residuals lift ARC-Easy by +2.2 at a BLiMP tie (SE ≈ 0.14) and a slightly lower
perplexity** — recovering part of the reasoning gap to the 125–143M leaderboard models,
at negligible parameter cost. Evaluated with EleutherAI `lm-evaluation-harness`.
## ⚠️ Loading — requires `trust_remote_code`
Value residuals are a **custom attention pathway**, so this model must be loaded with the
included modeling file. Loading it as a stock Qwen3 **silently drops the residual** and
understates quality.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R1")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R1", trust_remote_code=True)
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
```
## Architecture & training
Identical to R0 except the value-residual pathway: Qwen3-arch, **23 layers × 576**,
GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152),
z-loss 1e-4. Trained from scratch on ~8B tokens of FineWeb-Edu (sample-10BT), cosine LR
1.5e-3→1.5e-4, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO 4500 Blackwell.
## Intended use & limitations
Research **base** model (not instruction-tuned). Fluent English continuation and
reasonable recall for its size; limited multi-step reasoning, occasional repetition,
English-only. Not for production.
## Links
- Baseline: [altslate/JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
- Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu
## Paper
This model is described in [*Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M*](https://arxiv.org/abs/2609.14715)
(arXiv:2609.14715 · [HF Papers](https://huggingface.co/papers/2609.14715)).
```bibtex
@misc{rajput2026jugnu,
  title         = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
  author        = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
  year          = {2026},
  eprint        = {2609.14715},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.14715}
}
```

	---
	license: apache-2.0
	language:
	- en
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- tiny-lm
	- qwen3
	- value-residual
	- resformer
	- ablation
	datasets:
	- HuggingFaceFW/fineweb-edu
	---

	# JugnuLM-110M-R1 (value residuals)

	What "R1" means. This is rung 1 of the JugnuLM phase-2 ablation ladder. We take
	our honest 110M baseline — [JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
	(rung R0) — and add one architectural change, value residuals (ResFormer),
	holding everything else fixed (same data, tokens, schedule, batch, seed policy). The
	ladder keeps a lever only if it beats the previous rung. R1 does, so it's kept.

	## What value residuals are

	Each attention layer's value gains a **learned-gated residual from the first layer's
	value**:

	> `v_i = v_proj_i(x) + λ_i · v₀` (v₀ = layer-0 value; λ_i learned per layer)

	This gives deep layers direct access to early-layer content (Zhou et al., *Value Residual
	Learning*, ACL 2025 — [arXiv:2410.17897](https://arxiv.org/abs/2410.17897)). The 22 learned
	gates converge to non-trivial values (range −2.5 to +3.2), i.e. the path is genuinely used.

	## Result vs the R0 baseline

	\| 110M variant \| BLiMP ↑ \| ARC-Easy ↑ \| WikiText-2 byte-ppl ↓ \|
	\|---\|---\|---\|---\|
	\| R0 — baseline \| 81.25 \| 52.48 \| 1.95 \|
	\| R1 — + value residuals \| 81.10 \| 54.67 \| 1.94 \|

	**Value residuals lift ARC-Easy by +2.2 at a BLiMP tie (SE ≈ 0.14) and a slightly lower
	perplexity** — recovering part of the reasoning gap to the 125–143M leaderboard models,
	at negligible parameter cost. Evaluated with EleutherAI `lm-evaluation-harness`.

	## ⚠️ Loading — requires `trust_remote_code`

	Value residuals are a custom attention pathway, so this model must be loaded with the
	included modeling file. Loading it as a stock Qwen3 silently drops the residual and
	understates quality.

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer
	tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R1")
	model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R1", trust_remote_code=True)
	ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
	print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
	```

	## Architecture & training

	Identical to R0 except the value-residual pathway: Qwen3-arch, 23 layers × 576,
	GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152),
	z-loss 1e-4. Trained from scratch on ~8B tokens of FineWeb-Edu (sample-10BT), cosine LR
	1.5e-3→1.5e-4, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO 4500 Blackwell.

	## Intended use & limitations

	Research base model (not instruction-tuned). Fluent English continuation and
	reasonable recall for its size; limited multi-step reasoning, occasional repetition,
	English-only. Not for production.

	## Links

	- Baseline: [altslate/JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
	- Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu

	## Paper

	This model is described in [Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M](https://arxiv.org/abs/2609.14715)
	(arXiv:2609.14715 · [HF Papers](https://huggingface.co/papers/2609.14715)).

	```bibtex
	@misc{rajput2026jugnu,
	title = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
	author = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
	year = {2026},
	eprint = {2609.14715},
	archivePrefix = {arXiv},
	primaryClass = {cs.AI},
	url = {https://arxiv.org/abs/2609.14715}
	}
	```