Simplify model card

README.md

@@ -12,7 +12,6 @@ tags:
   - pretrained
   - hybrid
   - gated-deltanet
-  - sub-100m
 datasets:
   - nvidia/Nemotron-ClimbMix
   - IFM/TxT360-v2
@@ -31,111 +30,64 @@ datasets:
 
 # KLM 1.5
 
-**KLM 1.5 is a 99.0M-parameter base language model trained from scratch.** It scores 30.26 on the [Open SLM leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard)'s Intelligence Index. That is the highest of any model under 100M parameters, 6.3 points ahead of void.1 (23.92), and second under 150M, behind GPT-X3-150M (33.05). It also outscores LFM-2.5-230M-Base (23.47), which has 2.3 times as many parameters.
+KLM 1.5 is a 99M-parameter base language model, trained from scratch on 100B tokens. Most of its layers are recurrent Gated DeltaNet-2 layers, with full attention in every fourth layer.
 
-## What's new in 1.5
+## What's new
 
-[KLM 1.1](https://huggingface.co/MinimaLabs/KLM-1.1) is KeyLM-75M trained further: a 24-layer Llama/Qwen3-style transformer, 512 wide, with full attention in every layer, untied embeddings and a 24,576-token vocabulary. Its embedding and output tables take 25M of its 88M parameters.
+- **Hybrid layers.** 18 of the 24 layers are [Gated DeltaNet-2](https://github.com/fla-org/flash-linear-attention) layers, which carry a fixed-size memory through the sequence. The other 6 use full attention.
+- **Block Attention Residuals** ([Kimi Team, 2026](https://arxiv.org/abs/2603.15031)). Each layer reads a learned mix of the embeddings and every earlier layer's output, instead of one running residual sum.
+- **Exact n-gram memory.** The 125,000 most frequent bigrams and 38,000 most frequent trigrams each have a learned vector, looked up exactly and fed in next to the token embeddings. It holds 5.4M parameters.
+- **A small tokenizer.** 4,100 tokens, with every number split into digits. Tied embeddings cost 2.1M parameters, so almost all of the model is layers.
 
-KLM 1.5 shares none of its weights and keeps only the 24 × 512 shape. The changes:
+## Benchmarks
 
-- **Hybrid layers.** 18 of the 24 layers are [Gated DeltaNet-2](https://github.com/fla-org/flash-linear-attention) layers, which read the sequence through a fixed-size recurrent state. Every fourth layer is full attention (8 query and 4 KV heads, QK-norm, a sigmoid output gate, and a value residual from the first attention layer).
-- **Block Attention Residuals** ([Kimi Team, 2026](https://arxiv.org/abs/2603.15031)). Each layer's input is a learned, input-dependent mix of the token embedding, the n-gram memory and the output of every earlier layer, in place of one running residual sum.
-- **Exact n-gram memory.** The 125,000 most frequent bigrams and 38,000 most frequent trigrams in the training data each get their own learned 32-number row, found by exact lookup with no hashing. The memory holds 5.4M parameters and feeds the network as one more residual source.
-- **A 4,100-token tokenizer**, min-spark's 4,096 tokens plus `<|im_start|>`, `<|im_end|>`, `<think>` and `</think>`. It splits every number into single digits. Embeddings are tied and take 2.1M parameters, so the parameters KLM 1.1 spent on its vocabulary go to the layers instead.
-- **Training from scratch** on 100B tokens of a new mix, after a superposition warm-up over 128B tokens, with the MuonEq-R optimizer.
+Zero-shot, length-normalized accuracy. ARC, HellaSwag and PIQA use lm-eval 0.4.12; ArithMark-3 uses the official [ArithMark-3.0](https://huggingface.co/datasets/AxiomicLabs/ArithMark-3.0) script. On the [Open SLM leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard), these scores give KLM 1.5 an Intelligence Index of 30.26, first among models under 100M parameters and second under 150M. Peer scores are from the leaderboard on 2026-10-22.
 
-## Evaluation
-
-All scores are zero-shot and length-normalized. ARC, HellaSwag and PIQA use lm-eval 0.4.12; ArithMark-3 uses the official [ArithMark-3.0](https://huggingface.co/datasets/AxiomicLabs/ArithMark-3.0) script. Peer scores are from the leaderboard on 2026-10-22.
-
-![Benchmark scores against the leading models under 150M](charts/benchmarks.svg)
+![Benchmark scores](charts/benchmarks.svg)
 
 | Model | Params | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | ArithMark-3 |
 |-------|--------|----------|---------------|-----------|------|-------------|
 | **KLM 1.5** | 99.0M | 59.68% | 32.68% | 43.74% | 68.55% | 48.20% |
-| [GPT-X3-150M](https://huggingface.co/AxiomicLabs/GPT-X3-150M) | 149M | 59.64% | 31.31% | **46.16%** | **71.33%** | **50.90%** |
-| [Boris-2-Preview-1](https://huggingface.co/opencerebral/Boris-2-Preview-1) | 144M | 58.38% | 29.95% | 45.23% | 69.21% | 37.60% |
-| [cagliostro-v3.5](https://huggingface.co/bench-labs/cagliostro-v3.5) | 146M | 53.75% | 29.35% | 43.41% | 68.06% | 45.30% |
+| [GPT-X3-150M](https://huggingface.co/AxiomicLabs/GPT-X3-150M) | 149M | 59.64% | 31.31% | 46.16% | 71.33% | 50.90% |
 | [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) | 135M | 58.63% | 29.69% | 43.22% | 68.44% | 39.20% |
-| [LFM-2.5-230M-Base](https://huggingface.co/LiquidAI/LFM2.5-230M-Base) | 230M | **59.72%** | **34.98%** | 39.29% | 63.38% | 36.60% |
+| [LFM-2.5-230M-Base](https://huggingface.co/LiquidAI/LFM2.5-230M-Base) | 230M | 59.72% | 34.98% | 39.29% | 63.38% | 36.60% |
 | [void.1](https://huggingface.co/dotlabs/void.1) | 90.1M | 47.31% | 28.16% | 38.68% | 67.46% | 44.80% |
 | [KLM 1.1](https://huggingface.co/MinimaLabs/KLM-1.1) | 88.1M | 44.36% | 25.17% | 33.65% | 64.80% | 35.80% |
 
 ## Training
 
-KLM 1.5 was trained from scratch in three phases:
-
-| Phase | Steps | Text read | What the model learns |
-|---|---|---|---|
-| Superposition warm-up | 70,000 | 128B tokens of ClimbMix | Predicting the next group of 7 tokens |
-| Stable | 304,000 | 79.7B tokens | Next-token prediction at full learning rate |
-| Decay | 76,000 | 19.9B tokens | Next-token prediction while the learning rate falls to zero |
-
-### Superposition warm-up
+Training used 2,048-token sequences, 128 per step, in three stages:
 
-The warm-up follows Token-Superposition Training ([Peng, Gigant & Quesnelle, 2026](https://arxiv.org/abs/2605.06546)). Each input position is the average of 7 consecutive token embeddings, and its n-gram memory input is the average of their 7 rows. The model predicts which 7 tokens make up the next group, ignoring their order. A step costs about the same as a normal step but covers 7 times as much text. The warm-up's ClimbMix text is separate from the ClimbMix used in normal training.
+1. **Superposition warm-up**, 70,000 steps ([Token-Superposition Training](https://arxiv.org/abs/2605.06546)). Each input position averages 7 consecutive tokens, and the model predicts the next 7 as an unordered group. This covers 128B tokens of ClimbMix for the compute of about 18B.
+2. **Stable**, 80B tokens of next-token prediction at a constant learning rate.
+3. **Decay**, 20B tokens on a mix with more math and Q&A, while the learning rate falls to zero (1−√t). Most of the benchmark gain came here: at 70B tokens the model scored 52.9% on ARC-Easy and 38.6% on HellaSwag.
 
-![Benchmark scores during training](charts/training.svg)
+Matrices in the layers use MuonEq-R (learning rate 0.01); everything else uses AdamW (3e-3). Precision is bf16, and attention, recurrence and n-gram lookups never cross document boundaries.
 
-| Tokens of normal training | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | ArithMark-3 |
-|---|---|---|---|---|---|
-| 35.7B | 54.92% | 29.10% | 38.46% | 67.25% | 42.30% |
-| 45.1B | 52.86% | 28.75% | 38.10% | 67.25% | 44.80% |
-| 56.9B | 55.05% | 27.82% | 38.51% | 66.00% | 41.60% |
-| 69.7B | 52.90% | 28.84% | 38.64% | 66.43% | 43.60% |
-| **99.6B (released)** | **59.68%** | **32.68%** | **43.74%** | **68.55%** | **48.20%** |
+| Data | Sources | Stable | Decay |
+|---|---|---:|---:|
+| Web text | [ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix), [TxT360-v2](https://huggingface.co/datasets/IFM/TxT360-v2), [UltraX](https://huggingface.co/datasets/openbmb/UltraX-Preview) | 56% | 26% |
+| Q&A | TxT360-v2 Q&A, [Dolmino](https://huggingface.co/datasets/allenai/dolma3_dolmino_mix-100B-1025) Q&A and flashcards | 13% | 23% |
+| Textbooks | [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus), [Cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia) WikiHow | 12% | 14% |
+| Math | [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2), [UltraData-Math](https://huggingface.co/datasets/openbmb/UltraData-Math), [Math-Reasoning](https://huggingface.co/datasets/IFM/Math-Reasoning), Dolmino TinyMATH | 11% | 20% |
+| Science web | Dolmino | 3% | 8% |
+| Commonsense | [DCLM-baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) filtered for physical commonsense, ATOMIC 2020, ANLI, GLUCOSE, CREAK, SciQ, QASC | 3% | 8% |
+| Reasoning rewrites | [Pretrain-Behaviors](https://huggingface.co/datasets/IFM/Pretrain-Behaviors) | 2% | 1% |
 
-At the constant learning rate, scores moved up and down by 1–3 points with no clear trend. The decay over the final 19.9B tokens added 6.8 points on ARC-Easy, 5.1 on HellaSwag and 4.6 on ArithMark-3 over the last checkpoint before it.
-
-| Field | Value |
-|---|---|
-| Steps | 450,000 at 128 sequences × 2,048 tokens |
-| Precision | bf16 autocast |
-| Optimizer | MuonEq-R (matrices in the mixers and FFNs, learning rate 0.01, momentum 0.95, weight decay 0.01) + AdamW (embedding, norms, gates, convolutions, learning rate 3e-3; n-gram rows, learning rate 1.5e-2; no weight decay) |
-| Learning-rate schedule | 2,000-step warmup, constant, then 1−√t decay to zero over the final 76,000 steps |
-| Objective | Next-token loss, plus a penalty that is zero unless a Gated DeltaNet-2 decay channel saturates and stops keeping any memory |
-| Gradient clipping | 1.0 |
-| Attention masking | Intra-document, in the attention layers, the recurrent layers and the n-gram lookups |
-
-### Data
-
-| Source | Stable | Decay | License |
-|---|---:|---:|---|
-| [ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix) web text, read from the [OptimalScale copy](https://huggingface.co/datasets/OptimalScale/ClimbMix) (also the whole warm-up) | 43% | 21% | CC BY-NC 4.0 |
-| [TxT360-v2](https://huggingface.co/datasets/IFM/TxT360-v2) web pages with reading-comprehension Q&A | 10% | 8% | CC BY 4.0 |
-| TxT360-v2 web text | 6% | | CC BY 4.0 |
-| [UltraX](https://huggingface.co/datasets/openbmb/UltraX-Preview) refined FineWeb-Edu | 7% | 5% | Apache-2.0 |
-| [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) synthetic textbooks | 12% | 12% | ODC-BY |
-| [Cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia) WikiHow-style tutorials | | 2% | Apache-2.0 |
-| Commonsense: [DCLM-baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) filtered for physical commonsense, plus ATOMIC 2020, ANLI, GLUCOSE, CREAK, SciQ and QASC | 3% | 8% | CC BY 4.0, CC BY-NC 4.0 (ANLI), CC BY-NC 3.0 (SciQ), MIT (CREAK) |
-| [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) problems with solutions | 4% | 8% | CC BY 4.0 |
-| [UltraData-Math](https://huggingface.co/datasets/openbmb/UltraData-Math) Q&A and textbook math | 3% | 3% | Apache-2.0 |
-| [Math-Reasoning](https://huggingface.co/datasets/IFM/Math-Reasoning) worked problems with every digit operation written out (`math-thinking-oss`) | 2% | 4% | Apache-2.0 |
-| Math-Reasoning textbook-style rewrites (`math-rewrite`) | 1% | 1% | Apache-2.0 |
-| [Dolmino](https://huggingface.co/datasets/allenai/dolma3_dolmino_mix-100B-1025) grade-school math dialogues (`tinymath-mind`) | 1% | 4% | ODC-BY |
-| Dolmino science web crawl | 3% | 8% | ODC-BY |
-| Dolmino Q&A: passages with questions, synthetic Q&A and multiple-choice flashcards | 3% | 15% | ODC-BY |
-| [Pretrain-Behaviors](https://huggingface.co/datasets/IFM/Pretrain-Behaviors) reasoning and format rewrites | 2% | 1% | Apache-2.0 |
-
-Every source was checked against the five evaluation sets: any document sharing a 13-word sequence with an evaluation item was dropped.
+Any document sharing a 13-word sequence with an evaluation item was dropped. ClimbMix, ANLI and SciQ are licensed for non-commercial use only; the other sources are under CC BY, ODC-BY, Apache-2.0 or MIT.
 
 ## Architecture
 
 | Field | Value |
 |---|---|
-| Parameters | 98,975,308 (93,563,476 network + 5,411,832 n-gram memory) |
-| Layers | 24, in six groups of three Gated DeltaNet-2 layers and one attention layer |
+| Parameters | 98,975,308, of which 5,411,832 are the n-gram memory |
+| Layers | 24: three Gated DeltaNet-2 layers, then one attention layer, six times |
 | Hidden size | 512 |
-| Gated DeltaNet-2 | 4 heads × 128, short convolution 4, low-rank decay and output gates |
-| Attention | 8 query heads, 4 KV heads (GQA), head size 64, QK-RMSNorm, sigmoid output gate, value residual |
+| Gated DeltaNet-2 | 4 heads × 128 |
+| Attention | 8 query heads, 4 KV heads, head size 64, RoPE, QK-norm, output gate |
 | Feed-forward | SwiGLU, hidden size 1,408 |
-| Residual stream | Block Attention Residuals, one block per layer |
-| N-gram memory | 125,000 bigram + 38,000 trigram rows × 32, exact lookup |
-| Position encoding | RoPE (theta 10,000) in the attention layers |
 | Context length | 2,048 tokens |
-| Vocabulary | 4,100-token byte-level BPE (embedding padded to 4,160 rows), digits split |
-| Embeddings | Tied input and output |
+| Vocabulary | 4,100 tokens, tied embeddings |
 
 ## Usage
 
@@ -150,20 +102,16 @@ model = AutoModelForCausalLM.from_pretrained(
 ).to("cuda")
 
 inputs = tokenizer("The three primary colors are", return_tensors="pt").to("cuda")
-outputs = model.generate(
-    **inputs, max_new_tokens=40, do_sample=True,
-    temperature=0.7, top_p=0.9, repetition_penalty=1.1,
-)
+outputs = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.7, top_p=0.9)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
 
 ## Limitations
 
-- **Base model only.** KLM 1.5 completes text. It is not instruction-tuned and does not hold a conversation. The tokenizer has ChatML and `<think>` tokens for later chat versions, but the model was not trained on chat data.
-- **Small model.** Its world knowledge is thin and its facts are often wrong. Do not rely on it for factual answers or math.
-- English only, with a 2,048-token context window.
-- No safety alignment. Add your own filtering before any user-facing use.
+- A base model: it completes text and does not follow instructions or chat.
+- Its knowledge is thin and its facts are often wrong.
+- English only, 2,048-token context, no safety alignment.
 
 ## License
 
-Apache-2.0. See [LICENSE](LICENSE). Training-data licenses are listed in the [Data](#data) table.
+Apache-2.0. See [LICENSE](LICENSE).