JugnuLM-110M

A 109.7M parameter, deep-and-thin language model pretrained from scratch on ~8B tokens of FineWeb-Edu. It is the larger sibling of JugnuLM-53M and the baseline rung of an ongoing ablation study on what improves sub-150M models. Jugnu (जुगनू) = firefly.

Same deliberately conventional recipe as the 53M — Qwen3-style decoder (GQA, RoPE, SwiGLU, RMSNorm) with built-in QK-Norm and a z-loss — but scaled to ~110M with a deep-thin geometry (23 layers × 576, per MobileLLM's depth-over-width finding).

Results & the 53M → 110M jump

Evaluated with EleutherAI lm-evaluation-harness:

Metric	JugnuLM-53M	JugnuLM-110M	Δ
BLiMP (acc)	78.14	81.25	+3.11
ARC-Easy (acc)	51.43	52.48	+1.05
WikiText-2 byte-perplexity ↓	2.04	1.95	−0.09

The 81.25 BLiMP essentially matches GPT-X2-125M (81.28) — a top score on the sub-150M Tiny-ML Leaderboard — at 12% fewer parameters and **9× fewer training tokens** (8B vs 75B). Depth + scale bought a clean, across-the-board gain over the 53M.

Reproduce:

lm_eval --model hf \
  --model_args pretrained=altslate/JugnuLM-110M,dtype=bfloat16 \
  --tasks blimp,arc_easy,wikitext --batch_size auto

Architecture

Params	109.7M
Base architecture	Qwen3 (Llama-style + built-in QK-Norm)
Layers	23 (deep-thin)
Hidden size	576
Heads / KV heads	9 / 3 (grouped-query attention)
Head dim	64
FFN intermediate	1536 (SwiGLU)
Context	2048
Position / Norm	RoPE / RMSNorm + QK-Norm
Embeddings	tied
Tokenizer	SmolLM2 (49,152 vocab)

Training

• Data: HuggingFaceFW/fineweb-edu (sample-10BT), ~8B tokens seen, 2048-token blocks
• Optimizer: AdamW (β 0.9/0.95, wd 0.1), grad clip 1.0
• Schedule: cosine, 2k warmup, peak LR 1.5e-3 → floor 1.5e-4
• Batch: ~0.5M tokens/step · Stability: QK-Norm + z-loss (1e-4)
• Hardware: 4× NVIDIA RTX PRO 4500 Blackwell, bf16, DDP + torch.compile

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M")
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))

Intended use & limitations

Research / leaderboard base model — not instruction-tuned or aligned. Fluent English continuation and reasonable factual recall for its size; limited multi-step reasoning, occasional repetition, English-only. Not for production.

Citation / attribution

Trained by AltSlate Labs. Recipe and training code: https://github.com/AltSlate-Labs/jugnu

Paper

This model is described in Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M (arXiv:2609.14715 · HF Papers).

@misc{rajput2026jugnu,
  title         = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
  author        = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
  year          = {2026},
  eprint        = {2609.14715},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.14715}
}