Instructions to use altslate/JugnuLM-110M-R3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use altslate/JugnuLM-110M-R3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="altslate/JugnuLM-110M-R3", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R3", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use altslate/JugnuLM-110M-R3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "altslate/JugnuLM-110M-R3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "altslate/JugnuLM-110M-R3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/altslate/JugnuLM-110M-R3

SGLang

How to use altslate/JugnuLM-110M-R3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "altslate/JugnuLM-110M-R3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "altslate/JugnuLM-110M-R3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "altslate/JugnuLM-110M-R3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "altslate/JugnuLM-110M-R3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use altslate/JugnuLM-110M-R3 with Docker Model Runner:
```
docker model run hf.co/altslate/JugnuLM-110M-R3
```

JugnuLM-110M-R3 / README.md

rajpdus

Model card: cite arXiv:2609.14715

30ab5e3 verified 11 days ago

preview code

Raw History Blame Contribute Delete

4.77 kB

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - tiny-lm
  - qwen3
  - value-residual
  - muon
  - data-mixture
  - ablation
datasets:
  - HuggingFaceFW/fineweb-edu
  - mlfoundations/dclm-baseline-1.0
  - HuggingFaceTB/finemath
---
# JugnuLM-110M-R3 (value residuals + Muon + data blend)

**What "R3" means.** This is **rung 3 of the JugnuLM phase-2 ablation ladder.** We take
rung **R2** — [JugnuLM-110M-R2](https://huggingface.co/altslate/JugnuLM-110M-R2) (value
residuals + Muon) — and change **one** thing: the training **data**. Instead of
single-source FineWeb-Edu, R3 trains on a **blend** (per-sequence weighted mix), holding
architecture, optimizer, schedule, tokens, batch, and seed fixed.

## What changed: the data blend

| source | weight | role |
|---|---|---|
| FineWeb-Edu (sample-10BT) | 55% | language + general knowledge |
| DCLM-baseline | 35% | diversity → broader distribution |
| FineMath-4plus | 10% | light math reasoning |

Sequences are drawn per-step from a weighted-random source (a shuffled blend, not a staged
curriculum). Validation stayed a **fixed FineWeb-Edu held-out set** for comparable curves.

## Result — an honest negative for ARC

| 110M rung | BLiMP ↑ | ARC-Easy ↑ | WikiText-2 byte-ppl ↓ |
|---|---|---|---|
| R0 — baseline | 81.25 | 52.48 | 1.95 |
| R1 — + value residuals | 81.10 | 54.67 | 1.94 |
| R2 — + Muon | 80.78 | **56.10** | 1.932 |
| **R3 — + data blend** | **81.79** | 53.62 | **1.9092** |

**R3 has the best BLiMP and best perplexity of any rung — but ARC-Easy *dropped* −2.48 vs
R2 (56.10 → 53.62).** Since ARC-Easy is our binding constraint, **R3 is *not* kept in the
ladder** (it does not beat R2 on the metric that gates leaderboard rank; its EFF is marginally
below R2's). We publish it anyway, as the ablation result and as our strongest grammar/perplexity
checkpoint.

### Why the blend helped BLiMP/perplexity but hurt ARC
FineWeb-Edu's **educational-quality filtering** is closely aligned with ARC-Easy's
grade-school science questions. Diluting it 100% → 55% with general web text (DCLM) and math
(FineMath) improved broad language modeling (grammar, perplexity) but **removed ARC-relevant
educational signal** — DCLM is diverse but less science-dense, and FineMath targets math
reasoning, not ARC science. The lesson: at this scale, **for ARC-Easy the educational
distribution matters more than raw diversity.** Evaluated with EleutherAI `lm-evaluation-harness`.

## ⚠️ Loading — requires `trust_remote_code`

Value residuals are a **custom attention pathway**, so this model must be loaded with the
included modeling file. Loading it as a stock Qwen3 **silently drops the residual** and
understates quality.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R3")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R3", trust_remote_code=True)
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
```

## Architecture & training

Identical architecture/optimizer to R2 (value residuals + Muon): Qwen3-arch, **23 layers ×
576**, GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152),
z-loss 1e-4. Trained from scratch on ~8.4B tokens of the 55/35/10 blend above, Muon (peak
2e-2) + AdamW (1.5e-3) on a shared cosine, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO
4500 Blackwell. Only the data differs from R2.

## Intended use & limitations

Research **base** model (not instruction-tuned). Best-in-family English fluency/grammar and
lowest perplexity, but weaker ARC-Easy reasoning than R1/R2. Occasional repetition,
English-only. Not for production.

## Links

- Previous rung (best ARC): [altslate/JugnuLM-110M-R2](https://huggingface.co/altslate/JugnuLM-110M-R2)
- Baseline: [altslate/JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
- Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu

## Paper

This model is described in [*Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M*](https://arxiv.org/abs/2609.14715)
(arXiv:2609.14715 · [HF Papers](https://huggingface.co/papers/2609.14715)).

```bibtex
@misc{rajput2026jugnu,
  title         = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
  author        = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
  year          = {2026},
  eprint        = {2609.14715},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.14715}
}
```

	---
	license: apache-2.0
	language:
	- en
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- tiny-lm
	- qwen3
	- value-residual
	- muon
	- data-mixture
	- ablation
	datasets:
	- HuggingFaceFW/fineweb-edu
	- mlfoundations/dclm-baseline-1.0
	- HuggingFaceTB/finemath
	---

	# JugnuLM-110M-R3 (value residuals + Muon + data blend)

	What "R3" means. This is rung 3 of the JugnuLM phase-2 ablation ladder. We take
	rung R2 — [JugnuLM-110M-R2](https://huggingface.co/altslate/JugnuLM-110M-R2) (value
	residuals + Muon) — and change one thing: the training data. Instead of
	single-source FineWeb-Edu, R3 trains on a blend (per-sequence weighted mix), holding
	architecture, optimizer, schedule, tokens, batch, and seed fixed.

	## What changed: the data blend

	\| source \| weight \| role \|
	\|---\|---\|---\|
	\| FineWeb-Edu (sample-10BT) \| 55% \| language + general knowledge \|
	\| DCLM-baseline \| 35% \| diversity → broader distribution \|
	\| FineMath-4plus \| 10% \| light math reasoning \|

	Sequences are drawn per-step from a weighted-random source (a shuffled blend, not a staged
	curriculum). Validation stayed a fixed FineWeb-Edu held-out set for comparable curves.

	## Result — an honest negative for ARC

	\| 110M rung \| BLiMP ↑ \| ARC-Easy ↑ \| WikiText-2 byte-ppl ↓ \|
	\|---\|---\|---\|---\|
	\| R0 — baseline \| 81.25 \| 52.48 \| 1.95 \|
	\| R1 — + value residuals \| 81.10 \| 54.67 \| 1.94 \|
	\| R2 — + Muon \| 80.78 \| 56.10 \| 1.932 \|
	\| R3 — + data blend \| 81.79 \| 53.62 \| 1.9092 \|

	*R3 has the best BLiMP and best perplexity of any rung — but ARC-Easy dropped* −2.48 vs
	R2 (56.10 → 53.62). Since ARC-Easy is our binding constraint, R3 is not kept in the
	ladder** (it does not beat R2 on the metric that gates leaderboard rank; its EFF is marginally
	below R2's). We publish it anyway, as the ablation result and as our strongest grammar/perplexity
	checkpoint.

	### Why the blend helped BLiMP/perplexity but hurt ARC
	FineWeb-Edu's educational-quality filtering is closely aligned with ARC-Easy's
	grade-school science questions. Diluting it 100% → 55% with general web text (DCLM) and math
	(FineMath) improved broad language modeling (grammar, perplexity) but **removed ARC-relevant
	educational signal** — DCLM is diverse but less science-dense, and FineMath targets math
	reasoning, not ARC science. The lesson: at this scale, **for ARC-Easy the educational
	distribution matters more than raw diversity.** Evaluated with EleutherAI `lm-evaluation-harness`.

	## ⚠️ Loading — requires `trust_remote_code`

	Value residuals are a custom attention pathway, so this model must be loaded with the
	included modeling file. Loading it as a stock Qwen3 silently drops the residual and
	understates quality.

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer
	tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R3")
	model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R3", trust_remote_code=True)
	ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
	print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
	```

	## Architecture & training

	Identical architecture/optimizer to R2 (value residuals + Muon): Qwen3-arch, **23 layers ×
	576**, GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152),
	z-loss 1e-4. Trained from scratch on ~8.4B tokens of the 55/35/10 blend above, Muon (peak
	2e-2) + AdamW (1.5e-3) on a shared cosine, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO
	4500 Blackwell. Only the data differs from R2.

	## Intended use & limitations

	Research base model (not instruction-tuned). Best-in-family English fluency/grammar and
	lowest perplexity, but weaker ARC-Easy reasoning than R1/R2. Occasional repetition,
	English-only. Not for production.

	## Links

	- Previous rung (best ARC): [altslate/JugnuLM-110M-R2](https://huggingface.co/altslate/JugnuLM-110M-R2)
	- Baseline: [altslate/JugnuLM-110M](https://huggingface.co/altslate/JugnuLM-110M)
	- Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu

	## Paper

	This model is described in [Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M](https://arxiv.org/abs/2609.14715)
	(arXiv:2609.14715 · [HF Papers](https://huggingface.co/papers/2609.14715)).

	```bibtex
	@misc{rajput2026jugnu,
	title = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
	author = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
	year = {2026},
	eprint = {2609.14715},
	archivePrefix = {arXiv},
	primaryClass = {cs.AI},
	url = {https://arxiv.org/abs/2609.14715}
	}
	```