MinimaLabs/KLM-1.6

KLM 1.6

KLM 1.6 is a 99M-parameter base language model that continues training from a checkpoint of our KLM 1.5 taken before its decay stage.

What's new

KLM 1.6 decays its learning rate over 100B tokens instead of 20B. Aside from that not much has changed, since this is an incremental update over KLM 1.5.

Benchmarks

All scores are zero-shot with length-normalized accuracy.

• ARC, HellaSwag and PIQA were scored using lm-eval 0.4.12.
• ArithMark-3 was scored using the official ArithMark-3.0 script.
• Peer scores are taken from the Open SLM leaderboard as of 2026-11-09.

On that leaderboard, KLM 1.6's Intelligence Index is 33.95 which puts it first among models under 100M, under 150M and under 250M parameters, and only two models of any size score higher, both of them around 360M parameters.

Each chart's axis starts at random-guess accuracy, which is 25% for most benchmarks and 50% for PIQA, so the height of a bar shows how far a model scores above chance.

Model	Params	ARC-Easy	ARC-Challenge	HellaSwag	PIQA	ArithMark-3
KLM 1.6	99.0M	61.07%	33.53%	45.28%	70.18%	55.90%
KLM 1.5	99.0M	59.68%	32.68%	43.74%	68.55%	48.20%
GPT-X3-150M	149M	59.64%	31.31%	46.16%	71.33%	50.90%
SmolLM2-135M	135M	58.63%	29.69%	43.22%	68.44%	39.20%
LFM-2.5-230M-Base	230M	59.72%	34.98%	39.29%	63.38%	36.60%
void.1	90.1M	49.32%	29.35%	39.46%	67.19%	43.70%

Training

KLM 1.6 was trained on 2,048-token sequences in three stages, with 128 sequences per step in the first two and 192 in the decay.

1. Superposition warm-up for 70,000 steps, using Token-Superposition Training. Each input position is the average of 7 consecutive tokens, and the model learns to predict the next 7 tokens as an unordered group. This lets the warm-up cover 128B tokens of ClimbMix for the compute of about 18B.
2. Stable training on 80B tokens of next-token prediction at a constant learning rate.
3. Decay on 100B tokens while the learning rate falls to zero along a 1−√t curve.

The weight matrices were trained with MuonEq-R at a learning rate of 0.01, and the embeddings, norms and gates with AdamW at 3e-3, in bf16. Each 2,048-token sequence packs several documents, and the model never looks across a document boundary since attention is masked there, and the recurrent state and n-gram history are reset.

The training data came from the following sources.

Data	Sources
Web text	ClimbMix, TxT360-v2, UltraX
Q&A	TxT360-v2 Q&A, Dolmino Q&A and flashcards
Textbooks	Cosmopedia v2, Cosmopedia WikiHow
Math	OpenMathInstruct-2, UltraData-Math, Math-Reasoning, Dolmino TinyMATH, TinyGSM-MIND and CraneMath
Science web	Dolmino
Commonsense	DCLM-baseline filtered for physical commonsense, ATOMIC 2020, ANLI, GLUCOSE, CREAK, SciQ, QASC
Reasoning rewrites	Pretrain-Behaviors

Architecture

Field	Value
Parameters	98,975,308, of which 5,411,832 are the n-gram memory
Layers	24, Gated DeltaNet-2 and attention at 3:1
Hidden size	512
Gated DeltaNet-2	4 heads × 128
Attention	8 query heads, 4 KV heads, head size 64, RoPE, QK-norm, output gate
Residuals	Block Attention Residuals
N-gram memory	125,000 bigrams and 38,000 trigrams, each with its own learned vector
Feed-forward	SwiGLU, hidden size 1,408
Context length	2,048 tokens
Vocabulary	4,100 tokens, tied embeddings

Intended use

KLM 1.6 is a base model. It continues text but has not been trained to follow instructions or chat, so it is meant to be fine-tuned for your own use case. The tokenizer already has <|im_start|>, <|im_end|>, <think> and </think> as single tokens, so they are ready to use in chat and reasoning fine-tunes.

Usage

The model code is plain PyTorch, so KLM 1.6 runs on a CPU as well as on any GPU without extra packages beyond Transformers.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MinimaLabs/KLM-1.6"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to(device)

inputs = tokenizer("The three primary colors are", return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• The model's knowledge is thin, and the facts it states are often wrong.
• It only works in English and has a 2,048-token context window.
• It has no safety alignment, so add your own filtering before putting it in front of users.

License

KLM 1.6 is released under the Apache-2.0 license, which you can find in LICENSE.