| --- |
| license: cc-by-4.0 |
| language: |
| - en |
| - pl |
| library_name: custom |
| pipeline_tag: text-generation |
| tags: |
| - pytorch |
| - bdh |
| - fast-weights |
| - scaling |
| - language-model |
| base_model: pathwaycom/bdh |
| datasets: |
| - SlayerLab/research-mix-v1 |
| --- |
| |
| # BDH-150M-EN — Byte-Level BDH Scaling Study Checkpoint |
|
|
| A **150M-parameter BDH (Fast Weight Layers)** model trained to completion (30k steps) on a 970M-token byte-level research mix (80% EN fineweb-edu / 20% PL speakleash). This is the largest rung of the SlayerLab BDH scaling ladder. |
|
|
| ## Training summary |
|
|
| | Metric | Value | |
| |---|---| |
| | Params | ~151M | |
| | Steps | 30,000 (completed) | |
| | Seq len | 2048 (byte-level) | |
| | Data | research-mix-v1, 970M tokens, byte-level (vocab 256, no tokenizer) | |
| | Optimizer | AdamW lr=1e-3 → 1e-4 cosine, wd 0.1, warmup 1000 | |
| | Grad clip | ZClip | |
| | **Final val loss** | **1.099** | |
| | Hardware | RunPod A4000, single GPU | |
|
|
| ## Learning curve (val loss per 500 steps, from step 15500) |
|
|
| ``` |
| step val_loss |
| 15500 1.233 |
| 16000 1.325 |
| 16500 1.115 |
| 17000 1.273 |
| 17500 1.460 |
| 18000 1.169 |
| 18500 1.250 |
| 19000 1.258 |
| 19500 1.270 |
| 20000 1.260 |
| 20500 1.203 |
| 21000 1.175 |
| 21500 1.152 |
| 22000 1.241 |
| 22500 1.402 |
| 23000 1.199 |
| 23500 1.125 |
| 24000 1.079 |
| 24500 1.114 |
| 25000 1.215 |
| 25500 1.197 |
| 26000 1.286 |
| 26500 1.178 |
| 27000 1.390 |
| 27500 1.216 |
| 28000 1.531 |
| 28500 1.090 |
| 29000 1.115 |
| 29500 1.186 |
| 30000 1.099 |
| ``` |
|
|
| Full metrics (incl. steps 0–15000 from the pre-resume backup) in `bdh_150M_en.json`. |
|
|
| ## Architecture |
|
|
| - `BDH` from [pathwaycom/bdh](https://github.com/pathwaycom/bdh) — looped latent recurrence / fast weights |
| - `n_layer=8, n_embd=256, n_head=4, mlp_internal_dim_multiplier=768` |
| - Vocab: **byte-level (vocab 256)** — no tokenizer, works directly on UTF-8 bytes |
| - **~151M parameters** · seq length 2048 |
|
|
| ## Files |
|
|
| - `model.safetensors` — weights (state dict, `model.*` keys, compatible with `pathwaycom/bdh`) |
| - `config.json` — architecture + training config |
| - `bdh_150M_en.json` — full training metrics (loss, grad_norm per step) |
| - `bdh_150M_en.meta.json` — run metadata |
| |
| ## Load |
| |
| ```python |
| import sys, torch |
| sys.path.insert(0, "path/to/pathwaycom/bdh") |
| import bdh as bdh_mod |
| from safetensors.torch import load_file |
| |
| cfg = bdh_mod.BDHConfig( |
| n_layer=8, n_embd=256, n_head=4, |
| mlp_internal_dim_multiplier=768, dropout=0.1, |
| vocab_size=256, |
| ) |
| model = bdh_mod.BDH(cfg).bfloat16() |
| sd = load_file("model.safetensors") |
| model.load_state_dict({k.replace("model.", ""): v for k, v in sd.items()}, strict=False) |
| ``` |
| |
| ## Context |
|
|
| Part of the **BDH scaling study** (25M → 50M → 150M) — results & baselines comparison in [SlayerLab/bdh-scaling](https://huggingface.co/SlayerLab/bdh-scaling). Smaller rung: [SlayerLab/bdh-25m-pl](https://huggingface.co/SlayerLab/bdh-25m-pl). |
