README.md

---
license: cc-by-4.0
language:
- pl
library_name: custom
pipeline_tag: text-generation
tags:
- pytorch
- bdh
- fast-weights
- polish
- language-model
base_model: pathwaycom/bdh
datasets:
- SlayerLab/polish-dynaword-mix
---
# BDH-25M-PL — Polish Looped-Transformer (Fast-Weight) Language Model

A small, open-weight language model built on the **BDH (Fast Weight Layers)** architecture, trained on a clean Polish mix from **SlayerLab/polish-dynaword-mix** (100M tokens, byte-level).

## Abstract (EN)

**BDH-25M-PL** is a small, open-weight language model built on the **BDH (Fast Weight Layers)** architecture, drawn from the family of looped models with localized latent recurrence. Unlike a standard transformer, BDH shares its representation as both key and value (`Q==K`), so the internal state acts as a **working memory that adapts to context on the fly**, without changing weights. The model operates directly on UTF-8 bytes (vocab 256, no tokenizer) and was trained on a clean, diversified Polish corpus, `SlayerLab/polish-dynaword-mix` (100M tokens, legal content capped at ~7%). It is a **baseline**: it confirms the BDH architecture trains and generates correctly end-to-end, producing grammatically correct Polish sentences. Per scaling laws, a 25M-parameter model is undertrained for high quality (it would need ~0.5–2B tokens), yet it serves as a verifiable, public starting point for larger variants.

## Abstract (PL)

**BDH-25M-PL** to mały, otwarty model językowy oparty na architekturze **BDH (Fast Weight Layers, ang. warstwy szybkich wag)**, który wprowadzono w rodzinie modeli pętlowanych z lokalną rekurencją ukrytą (*looped latent recurrence*). W przeciwieństwie do klasycznego transformera, BDH współdzieli reprezentację jako klucz i wartość (`Q==K`), dzięki czemu wewnętrzny stan staje się swego rodzaju **pamięcią roboczą adaptującą się do kontekstu w locie**, bez zmiany wag. Model działa bezpośrednio na bajtach UTF-8 (vocab 256, brak tokenizera) i został wytrenowany na czystym, zróżnicowanym polskim zbiorze `SlayerLab/polish-dynaword-mix` (100 mln tokenów, treść prawna ograniczona do ~7%). Jest to **baseline**: potwierdza, że architektura BDH trenuje i generuje poprawnie end-to-end, generując gramatycznie poprawne polskie zdania. Zgodnie z prawami skalowania model o rozmiarze 25M param. jest niedoćwiczony dla wysokiej jakości (wymagałby ~0.5–2 mld tokenów), pełni jednak rolę weryfikowalnego, publicznego punktu startowego pod większe warianty.

## Training curves

Train and validation loss over 10,000 byte-level steps (final val loss ≈ 1.41; random-init baseline ≈ 5.6).

![Learning curves](learning_curve.png)

## Architecture

- `BDH` from [pathwaycom/bdh](https://github.com/pathwaycom/bdh) — looped latent recurrence / fast weights
- `n_layer=8, n_embd=256, n_head=4, mlp_internal_dim_multiplier=128`
- Vocab: **byte-level (vocab 256)** — no tokenizer, works directly on UTF-8 bytes
- **~25.3M parameters** · seq length 2048
- Key feature: `Q==K` (fast weights) — the shared state acts as both key and value (localized latent recurrence)

## Training

- **Data**: `SlayerLab/polish-dynaword-mix` (100M tokens, law cap 7%, dedup, cleaned)
- **Steps**: 10000 · **final val loss ≈ 1.41** · byte-level (random-init baseline ≈ 5.6)
- **Optimizer**: ZClip + protocol B.2 (as in the BDH paper)

## Note on scale

This is a **25M model trained on 100M tokens** — per scaling laws it is **undertrained for high quality**, but it serves as a working baseline: it generates grammatically-correct Polish and demonstrates that the BDH architecture trains and generates correctly end-to-end.

## Quick start

```python
import torch
from safetensors.torch import load_file
from bdh import BDH, BDHConfig

cfg = BDHConfig(n_layer=8, n_embd=256, n_head=4,
                mlp_internal_dim_multiplier=128, dropout=0.1, vocab_size=256)
model = BDH(cfg)
model.load_state_dict({k.replace("model.", ""): v
                       for k, v in load_file("model.safetensors").items()}, strict=True)
model.eval()

# byte-level prompt (no tokenizer)
prompt = "Warszawa jest stolicą Polski i "
ids = torch.tensor([list(prompt.encode("utf-8"))])
# ... autoregressive loop, one byte at a time with softmax(temp)
```

## Files

- `model.safetensors` — weights (101MB)
- `config.json` — hyperparameters and training metadata

## License

Weights: **CC-BY-4.0**. Architecture: MIT (pathwaycom/bdh).