File size: 2,926 Bytes
f7f02be
b447e73
 
 
 
 
 
 
 
f7f02be
b447e73
22f991b
 
b447e73
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
---
license: other
language:
- en
tags:
- text-generation
- merlin
- small-model
- pretraining
---

![f4ea02ff-a403-4fea-8f82-66f34dc040ad](https://cdn-uploads.huggingface.co/production/uploads/6a5f8c87a38cac087c2c6c05/TgW4BJvq49nFce0U940rh.jpeg)

# Merlin-9M

**Merlin-9M** is an **8.85M-parameter** decoder-only language model, trained
from scratch on **Stew-10B** (Wikipedia / Ultra-FineWeb / DCLM) with **no
benchmark data of any kind** in the training mix. It is the checkpoint from
step 12,502 of training, seen ~811M tokens.

## Architecture

| Property | Value |
|---|---|
| Parameters | 8,854,146 |
| `d_model` | 256 |
| Layers | 12 unique |
| Heads | 8 (head_dim 32) |
| FFN | SwiGLU(384) |
| Vocabulary | 8,192 (GPT-2's first 8192 merges) |
| Context | 1024 |
| RoPE | yes |
| Normalization | RMSNorm + QK-norm |
| Activation | SwiGLU |
| Output | logits softcap 15 |
| Optimizer | Muon (2D weights) + AdamW (embeddings) |

### Depth-recurrent core
Layers 3–8 are applied **twice**, giving **18 effective layers** from 12 unique
ones at zero extra parameter cost. The first pass uses a learned residual scale;
the second pass uses a smaller one (`scale_loop`).

### Multi-token prediction (MTP-2)
The training objective adds an auxiliary head predicting the token two positions
ahead, tied to the same embedding matrix. This does not affect inference — the
model is used as a standard causal LM.

## How to use

```python
from huggingface_hub import hf_hub_download
import torch

# download checkpoint
path = hf_hub_download(repo_id="Monster-Code/Merlin-9M",
                       filename="brasa9m_last.pt", repo_type="model")

# The checkpoint dict contains:
#   ck["model"]   -> state_dict
#   ck["cfg"]     -> architecture config
#   ck["step"]    -> training step (12502)

# See merlin9m.py in this repo for the model definition + a load_model() helper.
```

The `merlin9m.py` file in this repository is a self-contained inference module:

```python
from merlin9m import load_model, generate

model, cfg, tok, step = load_model("Monster-Code/Merlin-9M")
print(generate(model, cfg, tok, "The capital of France is"))
```

## Tokenizer

GPT-2 BPE. The model was trained on the **first 8,192 token IDs** of GPT-2's
50,257-token vocabulary — these are the most frequent merges in English. Input
token IDs outside `[0, 8191]` are clamped during inference.

## Training data

[Monster-Code/Stew-10B-tokenized](https://huggingface.co/datasets/Monster-Code/Stew-10B-tokenized)
— a 10B-token pretraining corpus:

| Source | Share | Tokens |
|---|---|---|
| Wikipedia (`20231101.en`) | 25% | 2.5B |
| Ultra-FineWeb (English) | 50% | 5.0B |
| DCLM-baseline 1.0 | 25% | 2.5B |

**No ARC, HellaSwag, PIQA, WinoGrande, or any other benchmark data was used
in training.**

## License

Mixed upstream licenses (CC BY-SA 3.0, GFDL, see upstream dataset cards).
No additional restrictions are imposed by this repository.