---
license: apache-2.0
language: [en, ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, tr, vi, zh]
library_name: pytorch
pipeline_tag: text-generation
tags: [dwslm, small-language-model, linear-attention, kda, from-scratch, base-model]
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceFW/fineweb-2
  - HuggingFaceCode/stack-v3-train
  - wikimedia/wikipedia
  - allenai/peS2o
  - open-web-math/open-web-math
  - PleIAs/English-PD
  - PleIAs/French-PD-Books
  - PleIAs/Russian-PD
  - PleIAs/US-PD-Newspapers
  - PleIAs/SEC
model-index:
  - name: DWSLM-256M-Base
    results:
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: HellaSwag, type: Rowan/hellaswag, split: validation}
        metrics:
          - {type: acc_norm, value: 0.3522, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: ARC-Easy, type: allenai/ai2_arc, split: test, config: ARC-Easy}
        metrics:
          - {type: acc_norm, value: 0.4444, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: ARC-Challenge, type: allenai/ai2_arc, split: test, config: ARC-Challenge}
        metrics:
          - {type: acc_norm, value: 0.2491, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: PIQA, type: ybisk/piqa, split: validation}
        metrics:
          - {type: acc_norm, value: 0.6137, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: WinoGrande, type: allenai/winogrande, split: validation, config: winogrande_xl}
        metrics:
          - {type: acc_norm, value: 0.5059, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: MMLU, type: cais/mmlu, split: test, config: all}
        metrics:
          - {type: acc_norm, value: 0.2723, name: length-normalized accuracy}
---

# DWSLM-256M-Base

The 4K-context base model: the end of DWSLM-256M's pretraining run (step 45,030, 23.6B tokens).

A small language model trained from scratch by [DWS Research](https://research.dws.rip/llm): our corpus, our tokenizer, our model code and training loop, no pretrained weights anywhere. This is a **base model**. It continues text; it was not instruction-tuned and has had no safety training. The full technical report is at [research.dws.rip/llm/report](https://research.dws.rip/llm/report).

| | |
|---|---|
| Parameters | 286M compute (252M in blocks, 34M head) + 369M lookup (34M token embedding, 336M value embeddings) = 655M stored |
| Architecture | 20 layers, d = 1024: five groups of three Kimi Delta Attention (linear attention) layers and one global attention layer |
| Context | 4,096 tokens |
| Training tokens | 23.61B (one epoch of a 23.2B-token corpus plus 289M rephrased tokens) |
| Tokenizer | DWSLMTok32.3, byte-level BPE, 32,768 tokens |
| Languages | English, 16 others (ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, tr, vi, zh), code |
| Weights | `model.safetensors`, fp32 (the training master weights) |
| License | Apache-2.0 (weights and code); training data keeps its own licenses, below |

Also available: [DWSLM-256M-Base-32K](https://huggingface.co/dwsllc/DWSLM-256M-Base-32K), this model extended to 32K tokens.

## Use

The architecture isn't in `transformers`, so this repo ships its own reference code in `dwslm/` (the training model code, copied verbatim). It needs an NVIDIA GPU: the linear-attention and convolution kernels come from [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Triton).

```bash
hf download dwsllc/DWSLM-256M-Base --local-dir DWSLM-256M-Base && cd DWSLM-256M-Base
pip install -r requirements.txt
python generate.py "The three primary colors are"
```

```python
from dwslm import load, next_logits, EOS_ID

model, tok = load(".")  # this folder
ids = [EOS_ID] + tok.encode("The capital of France is").ids  # every training document began with <eos>
logits = next_logits(model, [ids])  # (1, 32768) next-token logits
print(tok.decode([int(logits[0].argmax())]))
```

`next_logits` takes several sequences at once and packs them into one forward pass, each as its own segment. Scoring multiple-choice options works the same way: sum the log-probabilities of each option's tokens. There is no KV cache, so generation re-runs the prefix for every new token.

Notes:
- `config.json` sets `attn: flash` (PyTorch's FlashAttention-2 varlen path, Ampere or newer). Training used `cudnn`, which only runs on Hopper and Blackwell datacenter GPUs; both give the same attention.
- The tokenizer must keep `encode_special_tokens = True` (`load` sets it), so a literal `<eos>` in text is encoded as text, as it was in training.
- Fine-tuning: the weights are the fp32 masters. The training recipe used MuonH on matrices and AdamW elsewhere; any optimizer works, but use a learning rate near the end of pretraining's schedule (see the report).

## Architecture

- **Local layers (15):** Kimi Delta Attention, 8 heads × 128, a short causal convolution (kernel 4) on q/k/v, per-channel decay from a low-rank map, and a sigmoid-gated RMSNorm on the output. The state per head has a fixed size, so cost grows linearly with length.
- **Global layers (5, at 3, 7, 11, 15, 19):** grouped-query attention with 8 query and 2 KV heads, head dim 128, QK-norm, LeRoPE (learned per-band rotary scales), document-masked varlen attention and a per-head sigmoid output gate.
- **Residual stream:** block attention residuals. Each sublayer reads an 8-head softmax mix of the token embedding, every finished group's output and the current group's running sum; a learned λ adds the raw embedding back in. Branches are scaled by 8/20 (CompleteP).
- **Value embeddings:** ten token-indexed tables, one per even layer, gated per head into the attention values.
- **MLP:** ReLU², 1024 → 4096 → 1024. **Norm:** pre-RMSNorm. **Head:** untied, logits × 1/d, tanh softcap 20.

## Training data

15.7M documents from 18 openly licensed sources, 23.2B tokens, deduplicated with MinHash (181,089 near-duplicates removed) and filtered for OCR noise:

| type | tokens | sources (license) |
|---|---|---|
| web | 8.53B | FineWeb-Edu, FineWeb-2 (ODC-By 1.0) |
| code | 5.95B | The Stack v3, permissive licenses only (ODC-By 1.0) |
| books | 4.79B | PleIAs public-domain books in ten languages (public domain) |
| encyclopedia | 2.27B | Wikipedia, 16 languages (CC BY-SA 3.0) |
| science | 0.72B | peS2o (ODC-By 1.0) |
| government | 0.39B | PleIAs SEC filings (public domain) |
| news | 0.37B | PleIAs US public-domain newspapers (public domain) |
| math | 0.19B | OpenWebMath (ODC-By 1.0) |

Plus 289M tokens of synthetic rewrites (FAQ, tutorial, table, math, Q&A and dialogue formats) of FineWeb-Edu, peS2o and OpenWebMath documents, written by SmolLM2-1.7B-Instruct (Apache-2.0). About 0.1% of documents were held out for evaluation, and every training document sharing a rare 13-token sequence with an evaluation item was removed (2,802 documents).

## Training

| | |
|---|---|
| Steps | 45,030 × 524,288 tokens (one epoch) |
| Context ramp | 512 → 1K → 2K → 4K at 5%, 15% and 35% of tokens |
| Optimizer | MuonH (Hyperball) on matrices, AdamW elsewhere, no weight decay; MuonH peak LR 3.7 × 10⁻³ |
| Schedule | warmup-stable-decay: 1% warmup, linear decay over the last 20% to 5% of peak; high-quality sources upsampled 2× from 75% |
| Head | tied to the embedding until 2/3 of training, then untied |
| Precision | BF16 mixed precision, FP32 master weights |
| Compute | about 4.1 × 10¹⁹ FLOPs, 32.8 hours on one GPU |

## Evaluation

Zero-shot cloze, length-normalized accuracy (log-likelihood per UTF-8 byte of each option), full evaluation sets, every model scored by the same harness in BF16. Prompts follow lm-evaluation-harness / OLMES; MMLU uses SmolLM's cloze prompt.

| task | DWSLM-256M-Base | Gemma 3 270M | SmolLM2-360M | LFM2.5-350M | Qwen3.5-0.8B |
|---|---|---|---|---|---|
| HellaSwag | 0.352 | 0.415 | 0.564 | 0.453 | 0.537 |
| ARC-Easy | 0.444 | 0.579 | 0.671 | 0.699 | 0.625 |
| ARC-Challenge | 0.249 | 0.285 | 0.371 | 0.413 | 0.391 |
| PIQA | 0.614 | 0.688 | 0.725 | 0.675 | 0.710 |
| WinoGrande | 0.506 | 0.543 | 0.593 | 0.558 | 0.582 |
| MMLU | 0.272 | 0.299 | 0.351 | 0.349 | 0.350 |
| **average** | **0.406** | 0.468 | 0.546 | 0.524 | 0.532 |

Training tokens: 23.6B for this model, about 6T for Gemma 3 270M and about 4T for SmolLM2-360M. The six-task average has a standard error of about ±0.4 points. MMLU and ARC-Challenge are at or near chance (25%).

Held-out bits per byte on our corpus's held-out split: **0.9963** (Gemma 3 270M 1.101, SmolLM2-360M 1.167, Qwen3.5-0.8B 0.982 on the same text; the split comes from our own training distribution, and the baselines may have trained on these public documents, so read it as a sanity check, not a ranking).

## Limitations and risks

- A 256M base model: it is often wrong, and it states wrong things fluently. Don't use it for anything that matters.
- No instruction tuning, no safety training, no red-teaming; it will continue any text, including harmful or offensive text, and may reproduce biases in web and book data (the public-domain books are old).
- Half the corpus is not English, but quality is uneven across languages (Russian books score worst, at 2.57 bits per byte).
- Evaluated on cloze accuracy and bits per byte only so far.

## Citation

```bibtex
@misc{dwslm256m2026,
  title  = {DWSLM-256M: model card and technical report},
  author = {{DWS Research}},
  year   = {2026},
  url    = {https://research.dws.rip/llm/report}
}
```