DWSLM-256M release v1.0: weights, config, tokenizer, reference code, model card

ca76ef5 verified 1 day ago

9.75 kB

---
license: apache-2.0
language: [en, ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, tr, vi, zh]
library_name: pytorch
pipeline_tag: text-generation
tags: [dwslm, small-language-model, linear-attention, kda, from-scratch, base-model]
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceFW/fineweb-2
  - HuggingFaceCode/stack-v3-train
  - wikimedia/wikipedia
  - allenai/peS2o
  - open-web-math/open-web-math
  - PleIAs/English-PD
  - PleIAs/French-PD-Books
  - PleIAs/Russian-PD
  - PleIAs/US-PD-Newspapers
  - PleIAs/SEC
model-index:
  - name: DWSLM-256M-Base
    results:
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: HellaSwag, type: Rowan/hellaswag, split: validation}
        metrics:
          - {type: acc_norm, value: 0.3522, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: ARC-Easy, type: allenai/ai2_arc, split: test, config: ARC-Easy}
        metrics:
          - {type: acc_norm, value: 0.4444, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: ARC-Challenge, type: allenai/ai2_arc, split: test, config: ARC-Challenge}
        metrics:
          - {type: acc_norm, value: 0.2491, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: PIQA, type: ybisk/piqa, split: validation}
        metrics:
          - {type: acc_norm, value: 0.6137, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: WinoGrande, type: allenai/winogrande, split: validation, config: winogrande_xl}
        metrics:
          - {type: acc_norm, value: 0.5059, name: length-normalized accuracy}
      - task: {type: text-generation, name: zero-shot cloze}
        dataset: {name: MMLU, type: cais/mmlu, split: test, config: all}
        metrics:
          - {type: acc_norm, value: 0.2723, name: length-normalized accuracy}
---
# DWSLM-256M-Base

The 4K-context base model: the end of DWSLM-256M's pretraining run (step 45,030, 23.6B tokens).

A small language model trained from scratch by [DWS Research](https://research.dws.rip/llm): our corpus, our tokenizer, our model code and training loop, no pretrained weights anywhere. This is a **base model**. It continues text; it was not instruction-tuned and has had no safety training. The full technical report is at [research.dws.rip/llm/report](https://research.dws.rip/llm/report).

| | |
|---|---|
| Parameters | 286M compute (252M in blocks, 34M head) + 369M lookup (34M token embedding, 336M value embeddings) = 655M stored |
| Architecture | 20 layers, d = 1024: five groups of three Kimi Delta Attention (linear attention) layers and one global attention layer |
| Context | 4,096 tokens |
| Training tokens | 23.61B (one epoch of a 23.2B-token corpus plus 289M rephrased tokens) |
| Tokenizer | DWSLMTok32.3, byte-level BPE, 32,768 tokens |
| Languages | English, 16 others (ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, tr, vi, zh), code |
| Weights | `model.safetensors`, fp32 (the training master weights) |
| License | Apache-2.0 (weights and code); training data keeps its own licenses, below |

Also available: [DWSLM-256M-Base-32K](https://huggingface.co/dwsllc/DWSLM-256M-Base-32K), this model extended to 32K tokens.

## Use

The architecture isn't in `transformers`, so this repo ships its own reference code in `dwslm/` (the training model code, copied verbatim). It needs an NVIDIA GPU: the linear-attention and convolution kernels come from [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Triton).

```bash
hf download dwsllc/DWSLM-256M-Base --local-dir DWSLM-256M-Base && cd DWSLM-256M-Base
pip install -r requirements.txt
python generate.py "The three primary colors are"
```

```python
from dwslm import load, next_logits, EOS_ID
model, tok = load(".")  # this folder
ids = [EOS_ID] + tok.encode("The capital of France is").ids  # every training document began with <eos>
logits = next_logits(model, [ids])  # (1, 32768) next-token logits
print(tok.decode([int(logits[0].argmax())]))
```

`next_logits` takes several sequences at once and packs them into one forward pass, each as its own segment. Scoring multiple-choice options works the same way: sum the log-probabilities of each option's tokens. There is no KV cache, so generation re-runs the prefix for every new token.

Notes:
- `config.json` sets `attn: flash` (PyTorch's FlashAttention-2 varlen path, Ampere or newer). Training used `cudnn`, which only runs on Hopper and Blackwell datacenter GPUs; both give the same attention.
- The tokenizer must keep `encode_special_tokens = True` (`load` sets it), so a literal `<eos>` in text is encoded as text, as it was in training.
- Fine-tuning: the weights are the fp32 masters. The training recipe used MuonH on matrices and AdamW elsewhere; any optimizer works, but use a learning rate near the end of pretraining's schedule (see the report).

## Architecture

- **Local layers (15):** Kimi Delta Attention, 8 heads × 128, a short causal convolution (kernel 4) on q/k/v, per-channel decay from a low-rank map, and a sigmoid-gated RMSNorm on the output. The state per head has a fixed size, so cost grows linearly with length.
- **Global layers (5, at 3, 7, 11, 15, 19):** grouped-query attention with 8 query and 2 KV heads, head dim 128, QK-norm, LeRoPE (learned per-band rotary scales), document-masked varlen attention and a per-head sigmoid output gate.
- **Residual stream:** block attention residuals. Each sublayer reads an 8-head softmax mix of the token embedding, every finished group's output and the current group's running sum; a learned λ adds the raw embedding back in. Branches are scaled by 8/20 (CompleteP).
- **Value embeddings:** ten token-indexed tables, one per even layer, gated per head into the attention values.
- **MLP:** ReLU², 1024 → 4096 → 1024. **Norm:** pre-RMSNorm. **Head:** untied, logits × 1/d, tanh softcap 20.

## Training data

15.7M documents from 18 openly licensed sources, 23.2B tokens, deduplicated with MinHash (181,089 near-duplicates removed) and filtered for OCR noise:

| type | tokens | sources (license) |
|---|---|---|
| web | 8.53B | FineWeb-Edu, FineWeb-2 (ODC-By 1.0) |
| code | 5.95B | The Stack v3, permissive licenses only (ODC-By 1.0) |
| books | 4.79B | PleIAs public-domain books in ten languages (public domain) |
| encyclopedia | 2.27B | Wikipedia, 16 languages (CC BY-SA 3.0) |
| science | 0.72B | peS2o (ODC-By 1.0) |
| government | 0.39B | PleIAs SEC filings (public domain) |
| news | 0.37B | PleIAs US public-domain newspapers (public domain) |
| math | 0.19B | OpenWebMath (ODC-By 1.0) |

Plus 289M tokens of synthetic rewrites (FAQ, tutorial, table, math, Q&A and dialogue formats) of FineWeb-Edu, peS2o and OpenWebMath documents, written by SmolLM2-1.7B-Instruct (Apache-2.0). About 0.1% of documents were held out for evaluation, and every training document sharing a rare 13-token sequence with an evaluation item was removed (2,802 documents).

## Training

| | |
|---|---|
| Steps | 45,030 × 524,288 tokens (one epoch) |
| Context ramp | 512 → 1K → 2K → 4K at 5%, 15% and 35% of tokens |
| Optimizer | MuonH (Hyperball) on matrices, AdamW elsewhere, no weight decay; MuonH peak LR 3.7 × 10⁻³ |
| Schedule | warmup-stable-decay: 1% warmup, linear decay over the last 20% to 5% of peak; high-quality sources upsampled 2× from 75% |
| Head | tied to the embedding until 2/3 of training, then untied |
| Precision | BF16 mixed precision, FP32 master weights |
| Compute | about 4.1 × 10¹⁹ FLOPs, 32.8 hours on one GPU |

## Evaluation

Zero-shot cloze, length-normalized accuracy (log-likelihood per UTF-8 byte of each option), full evaluation sets, every model scored by the same harness in BF16. Prompts follow lm-evaluation-harness / OLMES; MMLU uses SmolLM's cloze prompt.

| task | DWSLM-256M-Base | Gemma 3 270M | SmolLM2-360M | LFM2.5-350M | Qwen3.5-0.8B |
|---|---|---|---|---|---|
| HellaSwag | 0.352 | 0.415 | 0.564 | 0.453 | 0.537 |
| ARC-Easy | 0.444 | 0.579 | 0.671 | 0.699 | 0.625 |
| ARC-Challenge | 0.249 | 0.285 | 0.371 | 0.413 | 0.391 |
| PIQA | 0.614 | 0.688 | 0.725 | 0.675 | 0.710 |
| WinoGrande | 0.506 | 0.543 | 0.593 | 0.558 | 0.582 |
| MMLU | 0.272 | 0.299 | 0.351 | 0.349 | 0.350 |
| **average** | **0.406** | 0.468 | 0.546 | 0.524 | 0.532 |

Training tokens: 23.6B for this model, about 6T for Gemma 3 270M and about 4T for SmolLM2-360M. The six-task average has a standard error of about ±0.4 points. MMLU and ARC-Challenge are at or near chance (25%).

Held-out bits per byte on our corpus's held-out split: **0.9963** (Gemma 3 270M 1.101, SmolLM2-360M 1.167, Qwen3.5-0.8B 0.982 on the same text; the split comes from our own training distribution, and the baselines may have trained on these public documents, so read it as a sanity check, not a ranking).

## Limitations and risks

- A 256M base model: it is often wrong, and it states wrong things fluently. Don't use it for anything that matters.
- No instruction tuning, no safety training, no red-teaming; it will continue any text, including harmful or offensive text, and may reproduce biases in web and book data (the public-domain books are old).
- Half the corpus is not English, but quality is uneven across languages (Russian books score worst, at 2.57 bits per byte).
- Evaluated on cloze accuracy and bits per byte only so far.

## Citation

```bibtex
@misc{dwslm256m2026,
  title  = {DWSLM-256M: model card and technical report},
  author = {{DWS Research}},
  year   = {2026},
  url    = {https://research.dws.rip/llm/report}
}
```

	---
	license: apache-2.0
	language: [en, ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, tr, vi, zh]
	library_name: pytorch
	pipeline_tag: text-generation
	tags: [dwslm, small-language-model, linear-attention, kda, from-scratch, base-model]
	datasets:
	- HuggingFaceFW/fineweb-edu
	- HuggingFaceFW/fineweb-2
	- HuggingFaceCode/stack-v3-train
	- wikimedia/wikipedia
	- allenai/peS2o
	- open-web-math/open-web-math
	- PleIAs/English-PD
	- PleIAs/French-PD-Books
	- PleIAs/Russian-PD
	- PleIAs/US-PD-Newspapers
	- PleIAs/SEC
	model-index:
	- name: DWSLM-256M-Base
	results:
	- task: {type: text-generation, name: zero-shot cloze}
	dataset: {name: HellaSwag, type: Rowan/hellaswag, split: validation}
	metrics:
	- {type: acc_norm, value: 0.3522, name: length-normalized accuracy}
	- task: {type: text-generation, name: zero-shot cloze}
	dataset: {name: ARC-Easy, type: allenai/ai2_arc, split: test, config: ARC-Easy}
	metrics:
	- {type: acc_norm, value: 0.4444, name: length-normalized accuracy}
	- task: {type: text-generation, name: zero-shot cloze}
	dataset: {name: ARC-Challenge, type: allenai/ai2_arc, split: test, config: ARC-Challenge}
	metrics:
	- {type: acc_norm, value: 0.2491, name: length-normalized accuracy}
	- task: {type: text-generation, name: zero-shot cloze}
	dataset: {name: PIQA, type: ybisk/piqa, split: validation}
	metrics:
	- {type: acc_norm, value: 0.6137, name: length-normalized accuracy}
	- task: {type: text-generation, name: zero-shot cloze}
	dataset: {name: WinoGrande, type: allenai/winogrande, split: validation, config: winogrande_xl}
	metrics:
	- {type: acc_norm, value: 0.5059, name: length-normalized accuracy}
	- task: {type: text-generation, name: zero-shot cloze}
	dataset: {name: MMLU, type: cais/mmlu, split: test, config: all}
	metrics:
	- {type: acc_norm, value: 0.2723, name: length-normalized accuracy}
	---

	# DWSLM-256M-Base

	The 4K-context base model: the end of DWSLM-256M's pretraining run (step 45,030, 23.6B tokens).

	A small language model trained from scratch by [DWS Research](https://research.dws.rip/llm): our corpus, our tokenizer, our model code and training loop, no pretrained weights anywhere. This is a base model. It continues text; it was not instruction-tuned and has had no safety training. The full technical report is at [research.dws.rip/llm/report](https://research.dws.rip/llm/report).

	\| \| \|
	\|---\|---\|
	\| Parameters \| 286M compute (252M in blocks, 34M head) + 369M lookup (34M token embedding, 336M value embeddings) = 655M stored \|
	\| Architecture \| 20 layers, d = 1024: five groups of three Kimi Delta Attention (linear attention) layers and one global attention layer \|
	\| Context \| 4,096 tokens \|
	\| Training tokens \| 23.61B (one epoch of a 23.2B-token corpus plus 289M rephrased tokens) \|
	\| Tokenizer \| DWSLMTok32.3, byte-level BPE, 32,768 tokens \|
	\| Languages \| English, 16 others (ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, tr, vi, zh), code \|
	\| Weights \| `model.safetensors`, fp32 (the training master weights) \|
	\| License \| Apache-2.0 (weights and code); training data keeps its own licenses, below \|

	Also available: [DWSLM-256M-Base-32K](https://huggingface.co/dwsllc/DWSLM-256M-Base-32K), this model extended to 32K tokens.

	## Use

	The architecture isn't in `transformers`, so this repo ships its own reference code in `dwslm/` (the training model code, copied verbatim). It needs an NVIDIA GPU: the linear-attention and convolution kernels come from [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Triton).

	```bash
	hf download dwsllc/DWSLM-256M-Base --local-dir DWSLM-256M-Base && cd DWSLM-256M-Base
	pip install -r requirements.txt
	python generate.py "The three primary colors are"
	```

	```python
	from dwslm import load, next_logits, EOS_ID

	model, tok = load(".") # this folder
	ids = [EOS_ID] + tok.encode("The capital of France is").ids # every training document began with <eos>
	logits = next_logits(model, [ids]) # (1, 32768) next-token logits
	print(tok.decode([int(logits[0].argmax())]))
	```

	`next_logits` takes several sequences at once and packs them into one forward pass, each as its own segment. Scoring multiple-choice options works the same way: sum the log-probabilities of each option's tokens. There is no KV cache, so generation re-runs the prefix for every new token.

	Notes:
	- `config.json` sets `attn: flash` (PyTorch's FlashAttention-2 varlen path, Ampere or newer). Training used `cudnn`, which only runs on Hopper and Blackwell datacenter GPUs; both give the same attention.
	- The tokenizer must keep `encode_special_tokens = True` (`load` sets it), so a literal `<eos>` in text is encoded as text, as it was in training.
	- Fine-tuning: the weights are the fp32 masters. The training recipe used MuonH on matrices and AdamW elsewhere; any optimizer works, but use a learning rate near the end of pretraining's schedule (see the report).

	## Architecture

	- Local layers (15): Kimi Delta Attention, 8 heads × 128, a short causal convolution (kernel 4) on q/k/v, per-channel decay from a low-rank map, and a sigmoid-gated RMSNorm on the output. The state per head has a fixed size, so cost grows linearly with length.
	- Global layers (5, at 3, 7, 11, 15, 19): grouped-query attention with 8 query and 2 KV heads, head dim 128, QK-norm, LeRoPE (learned per-band rotary scales), document-masked varlen attention and a per-head sigmoid output gate.
	- Residual stream: block attention residuals. Each sublayer reads an 8-head softmax mix of the token embedding, every finished group's output and the current group's running sum; a learned λ adds the raw embedding back in. Branches are scaled by 8/20 (CompleteP).
	- Value embeddings: ten token-indexed tables, one per even layer, gated per head into the attention values.
	- MLP: ReLU², 1024 → 4096 → 1024. Norm: pre-RMSNorm. Head: untied, logits × 1/d, tanh softcap 20.

	## Training data

	15.7M documents from 18 openly licensed sources, 23.2B tokens, deduplicated with MinHash (181,089 near-duplicates removed) and filtered for OCR noise:

	\| type \| tokens \| sources (license) \|
	\|---\|---\|---\|
	\| web \| 8.53B \| FineWeb-Edu, FineWeb-2 (ODC-By 1.0) \|
	\| code \| 5.95B \| The Stack v3, permissive licenses only (ODC-By 1.0) \|
	\| books \| 4.79B \| PleIAs public-domain books in ten languages (public domain) \|
	\| encyclopedia \| 2.27B \| Wikipedia, 16 languages (CC BY-SA 3.0) \|
	\| science \| 0.72B \| peS2o (ODC-By 1.0) \|
	\| government \| 0.39B \| PleIAs SEC filings (public domain) \|
	\| news \| 0.37B \| PleIAs US public-domain newspapers (public domain) \|
	\| math \| 0.19B \| OpenWebMath (ODC-By 1.0) \|

	Plus 289M tokens of synthetic rewrites (FAQ, tutorial, table, math, Q&A and dialogue formats) of FineWeb-Edu, peS2o and OpenWebMath documents, written by SmolLM2-1.7B-Instruct (Apache-2.0). About 0.1% of documents were held out for evaluation, and every training document sharing a rare 13-token sequence with an evaluation item was removed (2,802 documents).

	## Training

	\| \| \|
	\|---\|---\|
	\| Steps \| 45,030 × 524,288 tokens (one epoch) \|
	\| Context ramp \| 512 → 1K → 2K → 4K at 5%, 15% and 35% of tokens \|
	\| Optimizer \| MuonH (Hyperball) on matrices, AdamW elsewhere, no weight decay; MuonH peak LR 3.7 × 10⁻³ \|
	\| Schedule \| warmup-stable-decay: 1% warmup, linear decay over the last 20% to 5% of peak; high-quality sources upsampled 2× from 75% \|
	\| Head \| tied to the embedding until 2/3 of training, then untied \|
	\| Precision \| BF16 mixed precision, FP32 master weights \|
	\| Compute \| about 4.1 × 10¹⁹ FLOPs, 32.8 hours on one GPU \|

	## Evaluation

	Zero-shot cloze, length-normalized accuracy (log-likelihood per UTF-8 byte of each option), full evaluation sets, every model scored by the same harness in BF16. Prompts follow lm-evaluation-harness / OLMES; MMLU uses SmolLM's cloze prompt.

	\| task \| DWSLM-256M-Base \| Gemma 3 270M \| SmolLM2-360M \| LFM2.5-350M \| Qwen3.5-0.8B \|
	\|---\|---\|---\|---\|---\|---\|
	\| HellaSwag \| 0.352 \| 0.415 \| 0.564 \| 0.453 \| 0.537 \|
	\| ARC-Easy \| 0.444 \| 0.579 \| 0.671 \| 0.699 \| 0.625 \|
	\| ARC-Challenge \| 0.249 \| 0.285 \| 0.371 \| 0.413 \| 0.391 \|
	\| PIQA \| 0.614 \| 0.688 \| 0.725 \| 0.675 \| 0.710 \|
	\| WinoGrande \| 0.506 \| 0.543 \| 0.593 \| 0.558 \| 0.582 \|
	\| MMLU \| 0.272 \| 0.299 \| 0.351 \| 0.349 \| 0.350 \|
	\| average \| 0.406 \| 0.468 \| 0.546 \| 0.524 \| 0.532 \|

	Training tokens: 23.6B for this model, about 6T for Gemma 3 270M and about 4T for SmolLM2-360M. The six-task average has a standard error of about ±0.4 points. MMLU and ARC-Challenge are at or near chance (25%).

	Held-out bits per byte on our corpus's held-out split: 0.9963 (Gemma 3 270M 1.101, SmolLM2-360M 1.167, Qwen3.5-0.8B 0.982 on the same text; the split comes from our own training distribution, and the baselines may have trained on these public documents, so read it as a sanity check, not a ranking).

	## Limitations and risks

	- A 256M base model: it is often wrong, and it states wrong things fluently. Don't use it for anything that matters.
	- No instruction tuning, no safety training, no red-teaming; it will continue any text, including harmful or offensive text, and may reproduce biases in web and book data (the public-domain books are old).
	- Half the corpus is not English, but quality is uneven across languages (Russian books score worst, at 2.57 bits per byte).
	- Evaluated on cloze accuracy and bits per byte only so far.

	## Citation

	```bibtex
	@misc{dwslm256m2026,
	title = {DWSLM-256M: model card and technical report},
	author = {{DWS Research}},
	year = {2026},
	url = {https://research.dws.rip/llm/report}
	}
	```