Monster-Code commited on
Commit
b447e73
·
verified ·
1 Parent(s): f7f02be

Initial upload: Merlin-9M checkpoint (step 12502, 8.85M params)

Browse files
README.md CHANGED
@@ -1,3 +1,96 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: other
3
+ language:
4
+ - en
5
+ tags:
6
+ - text-generation
7
+ - merlin
8
+ - small-model
9
+ - pretraining
10
  ---
11
+
12
+ # Merlin-9M
13
+
14
+ **Merlin-9M** is an **8.85M-parameter** decoder-only language model, trained
15
+ from scratch on **Stew-10B** (Wikipedia / Ultra-FineWeb / DCLM) with **no
16
+ benchmark data of any kind** in the training mix. It is the checkpoint from
17
+ step 12,502 of training, seen ~811M tokens.
18
+
19
+ ## Architecture
20
+
21
+ | Property | Value |
22
+ |---|---|
23
+ | Parameters | 8,854,146 |
24
+ | `d_model` | 256 |
25
+ | Layers | 12 unique |
26
+ | Heads | 8 (head_dim 32) |
27
+ | FFN | SwiGLU(384) |
28
+ | Vocabulary | 8,192 (GPT-2's first 8192 merges) |
29
+ | Context | 1024 |
30
+ | RoPE | yes |
31
+ | Normalization | RMSNorm + QK-norm |
32
+ | Activation | SwiGLU |
33
+ | Output | logits softcap 15 |
34
+ | Optimizer | Muon (2D weights) + AdamW (embeddings) |
35
+
36
+ ### Depth-recurrent core
37
+ Layers 3–8 are applied **twice**, giving **18 effective layers** from 12 unique
38
+ ones at zero extra parameter cost. The first pass uses a learned residual scale;
39
+ the second pass uses a smaller one (`scale_loop`).
40
+
41
+ ### Multi-token prediction (MTP-2)
42
+ The training objective adds an auxiliary head predicting the token two positions
43
+ ahead, tied to the same embedding matrix. This does not affect inference — the
44
+ model is used as a standard causal LM.
45
+
46
+ ## How to use
47
+
48
+ ```python
49
+ from huggingface_hub import hf_hub_download
50
+ import torch
51
+
52
+ # download checkpoint
53
+ path = hf_hub_download(repo_id="Monster-Code/Merlin-9M",
54
+ filename="brasa9m_last.pt", repo_type="model")
55
+
56
+ # The checkpoint dict contains:
57
+ # ck["model"] -> state_dict
58
+ # ck["cfg"] -> architecture config
59
+ # ck["step"] -> training step (12502)
60
+
61
+ # See merlin9m.py in this repo for the model definition + a load_model() helper.
62
+ ```
63
+
64
+ The `merlin9m.py` file in this repository is a self-contained inference module:
65
+
66
+ ```python
67
+ from merlin9m import load_model, generate
68
+
69
+ model, cfg, tok, step = load_model("Monster-Code/Merlin-9M")
70
+ print(generate(model, cfg, tok, "The capital of France is"))
71
+ ```
72
+
73
+ ## Tokenizer
74
+
75
+ GPT-2 BPE. The model was trained on the **first 8,192 token IDs** of GPT-2's
76
+ 50,257-token vocabulary — these are the most frequent merges in English. Input
77
+ token IDs outside `[0, 8191]` are clamped during inference.
78
+
79
+ ## Training data
80
+
81
+ [Monster-Code/Stew-10B-tokenized](https://huggingface.co/datasets/Monster-Code/Stew-10B-tokenized)
82
+ — a 10B-token pretraining corpus:
83
+
84
+ | Source | Share | Tokens |
85
+ |---|---|---|
86
+ | Wikipedia (`20231101.en`) | 25% | 2.5B |
87
+ | Ultra-FineWeb (English) | 50% | 5.0B |
88
+ | DCLM-baseline 1.0 | 25% | 2.5B |
89
+
90
+ **No ARC, HellaSwag, PIQA, WinoGrande, or any other benchmark data was used
91
+ in training.**
92
+
93
+ ## License
94
+
95
+ Mixed upstream licenses (CC BY-SA 3.0, GFDL, see upstream dataset cards).
96
+ No additional restrictions are imposed by this repository.
__pycache__/merlin9m.cpython-313.pyc ADDED
Binary file (13.9 kB). View file
 
brasa9m_last.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0780508f7d14bcfd8b0e4b6a57b39603b233e717cc25eaa95d808bd531c04e7c
3
+ size 35452661
merlin9m.py ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Merlin-9M inference module.
3
+
4
+ A 8.85M-param decoder-only LM. Self-contained: no transformers needed to load it
5
+ (only for the GPT-2 tokenizer in the benchmark script).
6
+
7
+ Usage:
8
+ from merlin9m import load_model, generate
9
+ model, cfg, tok = load_model("Monster-Code/Merlin-9M")
10
+ print(generate(model, cfg, tok, "The capital of France is"))
11
+ """
12
+
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+
17
+ # ============================================================
18
+ # MODEL
19
+ # ============================================================
20
+ class RMSNorm(nn.Module):
21
+ def __init__(self, d, eps=1e-5):
22
+ super().__init__()
23
+ self.w = nn.Parameter(torch.ones(d))
24
+ self.eps = eps
25
+
26
+ def forward(self, x):
27
+ dt = x.dtype
28
+ x = x.float()
29
+ x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
30
+ return self.w * x.to(dt)
31
+
32
+
33
+ def build_rope(dim, max_len, device, base=10000.0):
34
+ inv = 1.0 / (base ** (torch.arange(0, dim, 2, device=device).float() / dim))
35
+ t = torch.arange(max_len, device=device).float()
36
+ f = torch.outer(t, inv)
37
+ f = torch.cat((f, f), dim=-1)
38
+ return torch.cos(f), torch.sin(f)
39
+
40
+
41
+ def rope_apply(x, cos, sin):
42
+ half = x.shape[-1] // 2
43
+ x1, x2 = x[..., :half], x[..., half:]
44
+ rx = torch.cat((-x2, x1), dim=-1)
45
+ return x * cos + rx * sin
46
+
47
+
48
+ class Block(nn.Module):
49
+ def __init__(self, d, nhead, ffn):
50
+ super().__init__()
51
+ self.d = d
52
+ self.nhead = nhead
53
+ self.hd = d // nhead
54
+ self.n1 = RMSNorm(d)
55
+ self.qkv = nn.Linear(d, 3 * d, bias=False)
56
+ self.o = nn.Linear(d, d, bias=False)
57
+ self.qk = RMSNorm(self.hd)
58
+ self.n2 = RMSNorm(d)
59
+ self.w1 = nn.Linear(d, ffn, bias=False)
60
+ self.w3 = nn.Linear(d, ffn, bias=False)
61
+ self.w2 = nn.Linear(ffn, d, bias=False)
62
+
63
+ def forward(self, x, cos, sin, scale):
64
+ b, t, d = x.shape
65
+ h = self.n1(x)
66
+ q, k, v = self.qkv(h).split(d, dim=-1)
67
+
68
+ q = q.view(b, t, self.nhead, self.hd).transpose(1, 2)
69
+ k = k.view(b, t, self.nhead, self.hd).transpose(1, 2)
70
+ v = v.view(b, t, self.nhead, self.hd).transpose(1, 2)
71
+
72
+ q = self.qk(q)
73
+ k = self.qk(k)
74
+ q = rope_apply(q, cos, sin)
75
+ k = rope_apply(k, cos, sin)
76
+
77
+ o = F.scaled_dot_product_attention(q, k, v, is_causal=True)
78
+ o = o.transpose(1, 2).reshape(b, t, d)
79
+ x = x + scale * self.o(o)
80
+
81
+ h = self.n2(x)
82
+ x = x + scale * self.w2(F.silu(self.w1(h)) * self.w3(h))
83
+ return x
84
+
85
+
86
+ class Brasa9M(nn.Module):
87
+ """12 unique blocks, 18 effective through a doubled 3..8 core."""
88
+
89
+ def __init__(self, cfg):
90
+ super().__init__()
91
+ d = cfg["d_model"]
92
+ self.cfg = cfg
93
+ self.emb = nn.Embedding(cfg["vocab"], d)
94
+ self.blocks = nn.ModuleList(
95
+ [Block(d, cfg["n_heads"], cfg["ffn"]) for _ in range(cfg["n_layers"])]
96
+ )
97
+ self.norm = RMSNorm(d)
98
+ self.mtp = nn.Linear(d, d, bias=False)
99
+ self.scale_first = nn.Parameter(torch.ones(1))
100
+ self.scale_loop = nn.Parameter(torch.ones(1) * 0.85)
101
+
102
+ cos, sin = build_rope(d // cfg["n_heads"], cfg["ctx"], "cpu")
103
+ self.register_buffer("rope_cos", cos, persistent=False)
104
+ self.register_buffer("rope_sin", sin, persistent=False)
105
+
106
+ def trunk(self, idx):
107
+ c = self.cfg
108
+ ctx = idx.shape[1]
109
+ cos = self.rope_cos[:ctx].to(idx.device).view(1, 1, ctx, -1)
110
+ sin = self.rope_sin[:ctx].to(idx.device).view(1, 1, ctx, -1)
111
+
112
+ x = self.emb(idx)
113
+ n = c["n_layers"]
114
+ pre = 3
115
+ core_end = 9
116
+ s1 = self.scale_first
117
+
118
+ for i in range(pre):
119
+ x = self.blocks[i](x, cos, sin, s1)
120
+ for _ in range(2):
121
+ for i in range(pre, core_end):
122
+ x = self.blocks[i](x, cos, sin, s1 if _ == 0 else self.scale_loop)
123
+ for i in range(core_end, n):
124
+ x = self.blocks[i](x, cos, sin, s1)
125
+ return self.norm(x)
126
+
127
+ def head(self, h):
128
+ logits = F.linear(h, self.emb.weight)
129
+ return self.cfg["softcap"] * torch.tanh(logits / self.cfg["softcap"])
130
+
131
+ def forward(self, idx, targets=None, targets2=None):
132
+ h = self.trunk(idx)
133
+ logits = self.head(h)
134
+ if targets is None:
135
+ return logits
136
+
137
+ logits = logits[:, : targets.shape[1]]
138
+ h = h[:, : targets.shape[1]]
139
+
140
+ loss1 = F.cross_entropy(
141
+ logits.reshape(-1, logits.size(-1)).float(), targets.reshape(-1),
142
+ ignore_index=-100,
143
+ )
144
+ lse = torch.logsumexp(logits.float(), dim=-1)
145
+ zl = self.cfg["z_loss"] * (lse ** 2).mean()
146
+
147
+ loss2 = torch.tensor(0.0, device=idx.device)
148
+ if targets2 is not None and self.cfg["mtp_weight"] > 0:
149
+ h2 = self.head(self.mtp(h))
150
+ h2 = h2[:, : targets2.shape[1]]
151
+ loss2 = F.cross_entropy(
152
+ h2.reshape(-1, h2.size(-1)).float(), targets2.reshape(-1),
153
+ ignore_index=-100,
154
+ )
155
+
156
+ loss = loss1 + self.cfg["mtp_weight"] * loss2 + zl
157
+ return loss, loss1.detach(), loss2.detach(), zl.detach()
158
+
159
+
160
+ # ============================================================
161
+ # LOADER
162
+ # ============================================================
163
+ def load_model(repo_id="Monster-Code/Merlin-9M", ckpt="brasa9m_last.pt",
164
+ device=None, token=None):
165
+ """Load config + weights from a HF repo. Returns (model, cfg, tokenizer)."""
166
+ from huggingface_hub import hf_hub_download
167
+ from transformers import AutoTokenizer
168
+
169
+ path = hf_hub_download(
170
+ repo_id=repo_id, filename=ckpt, repo_type="model", token=token
171
+ )
172
+ ck = torch.load(path, map_location="cpu", weights_only=False)
173
+ cfg = dict(ck["cfg"])
174
+
175
+ model = Brasa9M(cfg)
176
+ model.load_state_dict(ck["model"])
177
+ model.eval()
178
+
179
+ if device is None:
180
+ device = "cuda" if torch.cuda.is_available() else "cpu"
181
+ model = model.to(device)
182
+
183
+ # GPT-2 tokenizer. The model was trained on GPT-2's first 8192 ids.
184
+ tok = AutoTokenizer.from_pretrained("gpt2", token=token)
185
+
186
+ return model, cfg, tok, ck.get("step")
187
+
188
+
189
+ @torch.no_grad()
190
+ def generate(model, cfg, tok, prompt, max_new_tokens=40, temperature=0.8, top_k=50,
191
+ device=None):
192
+ if device is None:
193
+ device = next(model.parameters()).device
194
+
195
+ ids = torch.tensor([tok(prompt, add_special_tokens=False)["input_ids"]],
196
+ dtype=torch.long, device=device)
197
+ ids = ids.clamp(max=cfg["vocab"] - 1)
198
+ if ids.shape[1] < 2:
199
+ ids = ids.repeat(1, 2)
200
+
201
+ out = []
202
+ for _ in range(max_new_tokens):
203
+ inp = ids[:, -cfg["ctx"]:].clamp(max=cfg["vocab"] - 1)
204
+ logits = model(inp)[:, -1, :] / max(temperature, 1e-6)
205
+
206
+ if top_k:
207
+ k = min(top_k, logits.size(-1))
208
+ vals, idx = torch.topk(logits, k)
209
+ probs = F.softmax(vals, dim=-1)
210
+ pick = torch.multinomial(probs, 1)
211
+ nxt = idx[0, pick].item()
212
+ else:
213
+ nxt = logits.argmax(-1)[0].item()
214
+
215
+ out.append(nxt)
216
+ ids = torch.cat([ids, torch.tensor([[nxt]], device=device)], dim=1)
217
+
218
+ text = tok.decode(out, skip_special_tokens=True)
219
+ return text