DWSLM-256M-Base-32K

DWSLM-256M-Base extended to a 32K context: 953 more steps (500M tokens) on documents of at least 16K tokens, with YaRN ×8 over the learned rotary frequencies.

A small language model trained from scratch by DWS Research: our corpus, our tokenizer, our model code and training loop, no pretrained weights anywhere. This is a base model. It continues text; it was not instruction-tuned and has had no safety training. The full technical report is at research.dws.rip/llm/report.

Parameters	286M compute (252M in blocks, 34M head) + 369M lookup (34M token embedding, 336M value embeddings) = 655M stored
Architecture	20 layers, d = 1024: five groups of three Kimi Delta Attention (linear attention) layers and one global attention layer
Context	32,768 tokens (YaRN ×8 over a 4,096-token RoPE)
Training tokens	24.11B (DWSLM-256M-Base's 23.61B plus 0.50B at 32K)
Tokenizer	DWSLMTok32.3, byte-level BPE, 32,768 tokens
Languages	English, 16 others (ar, de, es, fr, hi, id, it, ja, ko, nl, pl, pt, ru, tr, vi, zh), code
Weights	model.safetensors, fp32 (the training master weights)
License	Apache-2.0 (weights and code); training data keeps its own licenses, below

Extended from DWSLM-256M-Base; everything below the evaluation section describes the shared pretraining run.

Use

The architecture isn't in transformers, so this repo ships its own reference code in dwslm/ (the training model code, copied verbatim). It needs an NVIDIA GPU: the linear-attention and convolution kernels come from flash-linear-attention (Triton).

hf download dwsllc/DWSLM-256M-Base-32K --local-dir DWSLM-256M-Base-32K && cd DWSLM-256M-Base-32K
pip install -r requirements.txt
python generate.py "The three primary colors are"

from dwslm import load, next_logits, EOS_ID

model, tok = load(".")  # this folder
ids = [EOS_ID] + tok.encode("The capital of France is").ids  # every training document began with <eos>
logits = next_logits(model, [ids])  # (1, 32768) next-token logits
print(tok.decode([int(logits[0].argmax())]))

next_logits takes several sequences at once and packs them into one forward pass, each as its own segment. Scoring multiple-choice options works the same way: sum the log-probabilities of each option's tokens. There is no KV cache, so generation re-runs the prefix for every new token.

Notes:

• config.json sets attn: flash (PyTorch's FlashAttention-2 varlen path, Ampere or newer). Training used cudnn, which only runs on Hopper and Blackwell datacenter GPUs; both give the same attention.
• The tokenizer must keep encode_special_tokens = True (load sets it), so a literal <eos> in text is encoded as text, as it was in training.
• Fine-tuning: the weights are the fp32 masters. The training recipe used MuonH on matrices and AdamW elsewhere; any optimizer works, but use a learning rate near the end of pretraining's schedule (see the report).

Architecture

• Local layers (15): Kimi Delta Attention, 8 heads × 128, a short causal convolution (kernel 4) on q/k/v, per-channel decay from a low-rank map, and a sigmoid-gated RMSNorm on the output. The state per head has a fixed size, so cost grows linearly with length.
• Global layers (5, at 3, 7, 11, 15, 19): grouped-query attention with 8 query and 2 KV heads, head dim 128, QK-norm, LeRoPE (learned per-band rotary scales), document-masked varlen attention and a per-head sigmoid output gate.
• Residual stream: block attention residuals. Each sublayer reads an 8-head softmax mix of the token embedding, every finished group's output and the current group's running sum; a learned λ adds the raw embedding back in. Branches are scaled by 8/20 (CompleteP).
• Value embeddings: ten token-indexed tables, one per even layer, gated per head into the attention values.
• MLP: ReLU², 1024 → 4096 → 1024. Norm: pre-RMSNorm. Head: untied, logits × 1/d, tanh softcap 20.

Training data

15.7M documents from 18 openly licensed sources, 23.2B tokens, deduplicated with MinHash (181,089 near-duplicates removed) and filtered for OCR noise:

type	tokens	sources (license)
web	8.53B	FineWeb-Edu, FineWeb-2 (ODC-By 1.0)
code	5.95B	The Stack v3, permissive licenses only (ODC-By 1.0)
books	4.79B	PleIAs public-domain books in ten languages (public domain)
encyclopedia	2.27B	Wikipedia, 16 languages (CC BY-SA 3.0)
science	0.72B	peS2o (ODC-By 1.0)
government	0.39B	PleIAs SEC filings (public domain)
news	0.37B	PleIAs US public-domain newspapers (public domain)
math	0.19B	OpenWebMath (ODC-By 1.0)

Plus 289M tokens of synthetic rewrites (FAQ, tutorial, table, math, Q&A and dialogue formats) of FineWeb-Edu, peS2o and OpenWebMath documents, written by SmolLM2-1.7B-Instruct (Apache-2.0). About 0.1% of documents were held out for evaluation, and every training document sharing a rare 13-token sequence with an evaluation item was removed (2,802 documents).

Training

Steps	45,030 × 524,288 tokens (one epoch)
Context ramp	512 → 1K → 2K → 4K at 5%, 15% and 35% of tokens
Optimizer	MuonH (Hyperball) on matrices, AdamW elsewhere, no weight decay; MuonH peak LR 3.7 × 10⁻³
Schedule	warmup-stable-decay: 1% warmup, linear decay over the last 20% to 5% of peak; high-quality sources upsampled 2× from 75%
Head	tied to the embedding until 2/3 of training, then untied
Precision	BF16 mixed precision, FP32 master weights
Compute	about 4.1 × 10¹⁹ FLOPs, 32.8 hours on one GPU
32K phase	953 steps × 524,288 tokens at 32,768 context on documents ≥ 16K tokens, YaRN ×8, learning rate from the base's final floor (1.85 × 10⁻⁴)

Evaluation

Zero-shot cloze, length-normalized accuracy (log-likelihood per UTF-8 byte of each option), full evaluation sets, every model scored by the same harness in BF16. Prompts follow lm-evaluation-harness / OLMES; MMLU uses SmolLM's cloze prompt.

task	DWSLM-256M-Base-32K	Gemma 3 270M	SmolLM2-360M	LFM2.5-350M	Qwen3.5-0.8B
HellaSwag	0.349	0.415	0.564	0.453	0.537
ARC-Easy	0.447	0.579	0.671	0.699	0.625
ARC-Challenge	0.251	0.285	0.371	0.413	0.391
PIQA	0.615	0.688	0.725	0.675	0.710
WinoGrande	0.507	0.543	0.593	0.558	0.582
MMLU	0.271	0.299	0.351	0.349	0.350
average	0.407	0.468	0.546	0.524	0.532

Training tokens: 24.1B for this model, about 6T for Gemma 3 270M and about 4T for SmolLM2-360M. The six-task average has a standard error of about ±0.4 points. MMLU and ARC-Challenge are at or near chance (25%).

Held-out bits per byte on our corpus's held-out split: 0.9957 at 4K, against 0.9963 before the extension: short-context quality held.

Bits per byte by position in the document, early in the 32K phase (step 95) and at its end:

positions	step 95	step 953
0 to 4K	0.831	0.822
4K to 8K	0.804	0.793
8K to 16K	0.806	0.795
16K to 32K	0.791	0.780

Limitations and risks

• A 256M base model: it is often wrong, and it states wrong things fluently. Don't use it for anything that matters.
• No instruction tuning, no safety training, no red-teaming; it will continue any text, including harmful or offensive text, and may reproduce biases in web and book data (the public-domain books are old).
• Half the corpus is not English, but quality is uneven across languages (Russian books score worst, at 2.57 bits per byte).
• Evaluated on cloze accuracy and bits per byte only so far.

Citation

@misc{dwslm256m2026,
  title  = {DWSLM-256M: model card and technical report},
  author = {{DWS Research}},
  year   = {2026},
  url    = {https://research.dws.rip/llm/report}
}