Instructions to use Monster-Code/Merlin-9M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Merlin
How to use Monster-Code/Merlin-9M with Merlin:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Initial upload: Merlin-9M checkpoint (step 12502, 8.85M params)
Browse files- README.md +94 -1
- __pycache__/merlin9m.cpython-313.pyc +0 -0
- brasa9m_last.pt +3 -0
- merlin9m.py +219 -0
README.md
CHANGED
|
@@ -1,3 +1,96 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: other
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
tags:
|
| 6 |
+
- text-generation
|
| 7 |
+
- merlin
|
| 8 |
+
- small-model
|
| 9 |
+
- pretraining
|
| 10 |
---
|
| 11 |
+
|
| 12 |
+
# Merlin-9M
|
| 13 |
+
|
| 14 |
+
**Merlin-9M** is an **8.85M-parameter** decoder-only language model, trained
|
| 15 |
+
from scratch on **Stew-10B** (Wikipedia / Ultra-FineWeb / DCLM) with **no
|
| 16 |
+
benchmark data of any kind** in the training mix. It is the checkpoint from
|
| 17 |
+
step 12,502 of training, seen ~811M tokens.
|
| 18 |
+
|
| 19 |
+
## Architecture
|
| 20 |
+
|
| 21 |
+
| Property | Value |
|
| 22 |
+
|---|---|
|
| 23 |
+
| Parameters | 8,854,146 |
|
| 24 |
+
| `d_model` | 256 |
|
| 25 |
+
| Layers | 12 unique |
|
| 26 |
+
| Heads | 8 (head_dim 32) |
|
| 27 |
+
| FFN | SwiGLU(384) |
|
| 28 |
+
| Vocabulary | 8,192 (GPT-2's first 8192 merges) |
|
| 29 |
+
| Context | 1024 |
|
| 30 |
+
| RoPE | yes |
|
| 31 |
+
| Normalization | RMSNorm + QK-norm |
|
| 32 |
+
| Activation | SwiGLU |
|
| 33 |
+
| Output | logits softcap 15 |
|
| 34 |
+
| Optimizer | Muon (2D weights) + AdamW (embeddings) |
|
| 35 |
+
|
| 36 |
+
### Depth-recurrent core
|
| 37 |
+
Layers 3–8 are applied **twice**, giving **18 effective layers** from 12 unique
|
| 38 |
+
ones at zero extra parameter cost. The first pass uses a learned residual scale;
|
| 39 |
+
the second pass uses a smaller one (`scale_loop`).
|
| 40 |
+
|
| 41 |
+
### Multi-token prediction (MTP-2)
|
| 42 |
+
The training objective adds an auxiliary head predicting the token two positions
|
| 43 |
+
ahead, tied to the same embedding matrix. This does not affect inference — the
|
| 44 |
+
model is used as a standard causal LM.
|
| 45 |
+
|
| 46 |
+
## How to use
|
| 47 |
+
|
| 48 |
+
```python
|
| 49 |
+
from huggingface_hub import hf_hub_download
|
| 50 |
+
import torch
|
| 51 |
+
|
| 52 |
+
# download checkpoint
|
| 53 |
+
path = hf_hub_download(repo_id="Monster-Code/Merlin-9M",
|
| 54 |
+
filename="brasa9m_last.pt", repo_type="model")
|
| 55 |
+
|
| 56 |
+
# The checkpoint dict contains:
|
| 57 |
+
# ck["model"] -> state_dict
|
| 58 |
+
# ck["cfg"] -> architecture config
|
| 59 |
+
# ck["step"] -> training step (12502)
|
| 60 |
+
|
| 61 |
+
# See merlin9m.py in this repo for the model definition + a load_model() helper.
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
The `merlin9m.py` file in this repository is a self-contained inference module:
|
| 65 |
+
|
| 66 |
+
```python
|
| 67 |
+
from merlin9m import load_model, generate
|
| 68 |
+
|
| 69 |
+
model, cfg, tok, step = load_model("Monster-Code/Merlin-9M")
|
| 70 |
+
print(generate(model, cfg, tok, "The capital of France is"))
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
## Tokenizer
|
| 74 |
+
|
| 75 |
+
GPT-2 BPE. The model was trained on the **first 8,192 token IDs** of GPT-2's
|
| 76 |
+
50,257-token vocabulary — these are the most frequent merges in English. Input
|
| 77 |
+
token IDs outside `[0, 8191]` are clamped during inference.
|
| 78 |
+
|
| 79 |
+
## Training data
|
| 80 |
+
|
| 81 |
+
[Monster-Code/Stew-10B-tokenized](https://huggingface.co/datasets/Monster-Code/Stew-10B-tokenized)
|
| 82 |
+
— a 10B-token pretraining corpus:
|
| 83 |
+
|
| 84 |
+
| Source | Share | Tokens |
|
| 85 |
+
|---|---|---|
|
| 86 |
+
| Wikipedia (`20231101.en`) | 25% | 2.5B |
|
| 87 |
+
| Ultra-FineWeb (English) | 50% | 5.0B |
|
| 88 |
+
| DCLM-baseline 1.0 | 25% | 2.5B |
|
| 89 |
+
|
| 90 |
+
**No ARC, HellaSwag, PIQA, WinoGrande, or any other benchmark data was used
|
| 91 |
+
in training.**
|
| 92 |
+
|
| 93 |
+
## License
|
| 94 |
+
|
| 95 |
+
Mixed upstream licenses (CC BY-SA 3.0, GFDL, see upstream dataset cards).
|
| 96 |
+
No additional restrictions are imposed by this repository.
|
__pycache__/merlin9m.cpython-313.pyc
ADDED
|
Binary file (13.9 kB). View file
|
|
|
brasa9m_last.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0780508f7d14bcfd8b0e4b6a57b39603b233e717cc25eaa95d808bd531c04e7c
|
| 3 |
+
size 35452661
|
merlin9m.py
ADDED
|
@@ -0,0 +1,219 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Merlin-9M inference module.
|
| 3 |
+
|
| 4 |
+
A 8.85M-param decoder-only LM. Self-contained: no transformers needed to load it
|
| 5 |
+
(only for the GPT-2 tokenizer in the benchmark script).
|
| 6 |
+
|
| 7 |
+
Usage:
|
| 8 |
+
from merlin9m import load_model, generate
|
| 9 |
+
model, cfg, tok = load_model("Monster-Code/Merlin-9M")
|
| 10 |
+
print(generate(model, cfg, tok, "The capital of France is"))
|
| 11 |
+
"""
|
| 12 |
+
|
| 13 |
+
import torch
|
| 14 |
+
import torch.nn as nn
|
| 15 |
+
import torch.nn.functional as F
|
| 16 |
+
|
| 17 |
+
# ============================================================
|
| 18 |
+
# MODEL
|
| 19 |
+
# ============================================================
|
| 20 |
+
class RMSNorm(nn.Module):
|
| 21 |
+
def __init__(self, d, eps=1e-5):
|
| 22 |
+
super().__init__()
|
| 23 |
+
self.w = nn.Parameter(torch.ones(d))
|
| 24 |
+
self.eps = eps
|
| 25 |
+
|
| 26 |
+
def forward(self, x):
|
| 27 |
+
dt = x.dtype
|
| 28 |
+
x = x.float()
|
| 29 |
+
x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 30 |
+
return self.w * x.to(dt)
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def build_rope(dim, max_len, device, base=10000.0):
|
| 34 |
+
inv = 1.0 / (base ** (torch.arange(0, dim, 2, device=device).float() / dim))
|
| 35 |
+
t = torch.arange(max_len, device=device).float()
|
| 36 |
+
f = torch.outer(t, inv)
|
| 37 |
+
f = torch.cat((f, f), dim=-1)
|
| 38 |
+
return torch.cos(f), torch.sin(f)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def rope_apply(x, cos, sin):
|
| 42 |
+
half = x.shape[-1] // 2
|
| 43 |
+
x1, x2 = x[..., :half], x[..., half:]
|
| 44 |
+
rx = torch.cat((-x2, x1), dim=-1)
|
| 45 |
+
return x * cos + rx * sin
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
class Block(nn.Module):
|
| 49 |
+
def __init__(self, d, nhead, ffn):
|
| 50 |
+
super().__init__()
|
| 51 |
+
self.d = d
|
| 52 |
+
self.nhead = nhead
|
| 53 |
+
self.hd = d // nhead
|
| 54 |
+
self.n1 = RMSNorm(d)
|
| 55 |
+
self.qkv = nn.Linear(d, 3 * d, bias=False)
|
| 56 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 57 |
+
self.qk = RMSNorm(self.hd)
|
| 58 |
+
self.n2 = RMSNorm(d)
|
| 59 |
+
self.w1 = nn.Linear(d, ffn, bias=False)
|
| 60 |
+
self.w3 = nn.Linear(d, ffn, bias=False)
|
| 61 |
+
self.w2 = nn.Linear(ffn, d, bias=False)
|
| 62 |
+
|
| 63 |
+
def forward(self, x, cos, sin, scale):
|
| 64 |
+
b, t, d = x.shape
|
| 65 |
+
h = self.n1(x)
|
| 66 |
+
q, k, v = self.qkv(h).split(d, dim=-1)
|
| 67 |
+
|
| 68 |
+
q = q.view(b, t, self.nhead, self.hd).transpose(1, 2)
|
| 69 |
+
k = k.view(b, t, self.nhead, self.hd).transpose(1, 2)
|
| 70 |
+
v = v.view(b, t, self.nhead, self.hd).transpose(1, 2)
|
| 71 |
+
|
| 72 |
+
q = self.qk(q)
|
| 73 |
+
k = self.qk(k)
|
| 74 |
+
q = rope_apply(q, cos, sin)
|
| 75 |
+
k = rope_apply(k, cos, sin)
|
| 76 |
+
|
| 77 |
+
o = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 78 |
+
o = o.transpose(1, 2).reshape(b, t, d)
|
| 79 |
+
x = x + scale * self.o(o)
|
| 80 |
+
|
| 81 |
+
h = self.n2(x)
|
| 82 |
+
x = x + scale * self.w2(F.silu(self.w1(h)) * self.w3(h))
|
| 83 |
+
return x
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
class Brasa9M(nn.Module):
|
| 87 |
+
"""12 unique blocks, 18 effective through a doubled 3..8 core."""
|
| 88 |
+
|
| 89 |
+
def __init__(self, cfg):
|
| 90 |
+
super().__init__()
|
| 91 |
+
d = cfg["d_model"]
|
| 92 |
+
self.cfg = cfg
|
| 93 |
+
self.emb = nn.Embedding(cfg["vocab"], d)
|
| 94 |
+
self.blocks = nn.ModuleList(
|
| 95 |
+
[Block(d, cfg["n_heads"], cfg["ffn"]) for _ in range(cfg["n_layers"])]
|
| 96 |
+
)
|
| 97 |
+
self.norm = RMSNorm(d)
|
| 98 |
+
self.mtp = nn.Linear(d, d, bias=False)
|
| 99 |
+
self.scale_first = nn.Parameter(torch.ones(1))
|
| 100 |
+
self.scale_loop = nn.Parameter(torch.ones(1) * 0.85)
|
| 101 |
+
|
| 102 |
+
cos, sin = build_rope(d // cfg["n_heads"], cfg["ctx"], "cpu")
|
| 103 |
+
self.register_buffer("rope_cos", cos, persistent=False)
|
| 104 |
+
self.register_buffer("rope_sin", sin, persistent=False)
|
| 105 |
+
|
| 106 |
+
def trunk(self, idx):
|
| 107 |
+
c = self.cfg
|
| 108 |
+
ctx = idx.shape[1]
|
| 109 |
+
cos = self.rope_cos[:ctx].to(idx.device).view(1, 1, ctx, -1)
|
| 110 |
+
sin = self.rope_sin[:ctx].to(idx.device).view(1, 1, ctx, -1)
|
| 111 |
+
|
| 112 |
+
x = self.emb(idx)
|
| 113 |
+
n = c["n_layers"]
|
| 114 |
+
pre = 3
|
| 115 |
+
core_end = 9
|
| 116 |
+
s1 = self.scale_first
|
| 117 |
+
|
| 118 |
+
for i in range(pre):
|
| 119 |
+
x = self.blocks[i](x, cos, sin, s1)
|
| 120 |
+
for _ in range(2):
|
| 121 |
+
for i in range(pre, core_end):
|
| 122 |
+
x = self.blocks[i](x, cos, sin, s1 if _ == 0 else self.scale_loop)
|
| 123 |
+
for i in range(core_end, n):
|
| 124 |
+
x = self.blocks[i](x, cos, sin, s1)
|
| 125 |
+
return self.norm(x)
|
| 126 |
+
|
| 127 |
+
def head(self, h):
|
| 128 |
+
logits = F.linear(h, self.emb.weight)
|
| 129 |
+
return self.cfg["softcap"] * torch.tanh(logits / self.cfg["softcap"])
|
| 130 |
+
|
| 131 |
+
def forward(self, idx, targets=None, targets2=None):
|
| 132 |
+
h = self.trunk(idx)
|
| 133 |
+
logits = self.head(h)
|
| 134 |
+
if targets is None:
|
| 135 |
+
return logits
|
| 136 |
+
|
| 137 |
+
logits = logits[:, : targets.shape[1]]
|
| 138 |
+
h = h[:, : targets.shape[1]]
|
| 139 |
+
|
| 140 |
+
loss1 = F.cross_entropy(
|
| 141 |
+
logits.reshape(-1, logits.size(-1)).float(), targets.reshape(-1),
|
| 142 |
+
ignore_index=-100,
|
| 143 |
+
)
|
| 144 |
+
lse = torch.logsumexp(logits.float(), dim=-1)
|
| 145 |
+
zl = self.cfg["z_loss"] * (lse ** 2).mean()
|
| 146 |
+
|
| 147 |
+
loss2 = torch.tensor(0.0, device=idx.device)
|
| 148 |
+
if targets2 is not None and self.cfg["mtp_weight"] > 0:
|
| 149 |
+
h2 = self.head(self.mtp(h))
|
| 150 |
+
h2 = h2[:, : targets2.shape[1]]
|
| 151 |
+
loss2 = F.cross_entropy(
|
| 152 |
+
h2.reshape(-1, h2.size(-1)).float(), targets2.reshape(-1),
|
| 153 |
+
ignore_index=-100,
|
| 154 |
+
)
|
| 155 |
+
|
| 156 |
+
loss = loss1 + self.cfg["mtp_weight"] * loss2 + zl
|
| 157 |
+
return loss, loss1.detach(), loss2.detach(), zl.detach()
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
# ============================================================
|
| 161 |
+
# LOADER
|
| 162 |
+
# ============================================================
|
| 163 |
+
def load_model(repo_id="Monster-Code/Merlin-9M", ckpt="brasa9m_last.pt",
|
| 164 |
+
device=None, token=None):
|
| 165 |
+
"""Load config + weights from a HF repo. Returns (model, cfg, tokenizer)."""
|
| 166 |
+
from huggingface_hub import hf_hub_download
|
| 167 |
+
from transformers import AutoTokenizer
|
| 168 |
+
|
| 169 |
+
path = hf_hub_download(
|
| 170 |
+
repo_id=repo_id, filename=ckpt, repo_type="model", token=token
|
| 171 |
+
)
|
| 172 |
+
ck = torch.load(path, map_location="cpu", weights_only=False)
|
| 173 |
+
cfg = dict(ck["cfg"])
|
| 174 |
+
|
| 175 |
+
model = Brasa9M(cfg)
|
| 176 |
+
model.load_state_dict(ck["model"])
|
| 177 |
+
model.eval()
|
| 178 |
+
|
| 179 |
+
if device is None:
|
| 180 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 181 |
+
model = model.to(device)
|
| 182 |
+
|
| 183 |
+
# GPT-2 tokenizer. The model was trained on GPT-2's first 8192 ids.
|
| 184 |
+
tok = AutoTokenizer.from_pretrained("gpt2", token=token)
|
| 185 |
+
|
| 186 |
+
return model, cfg, tok, ck.get("step")
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
@torch.no_grad()
|
| 190 |
+
def generate(model, cfg, tok, prompt, max_new_tokens=40, temperature=0.8, top_k=50,
|
| 191 |
+
device=None):
|
| 192 |
+
if device is None:
|
| 193 |
+
device = next(model.parameters()).device
|
| 194 |
+
|
| 195 |
+
ids = torch.tensor([tok(prompt, add_special_tokens=False)["input_ids"]],
|
| 196 |
+
dtype=torch.long, device=device)
|
| 197 |
+
ids = ids.clamp(max=cfg["vocab"] - 1)
|
| 198 |
+
if ids.shape[1] < 2:
|
| 199 |
+
ids = ids.repeat(1, 2)
|
| 200 |
+
|
| 201 |
+
out = []
|
| 202 |
+
for _ in range(max_new_tokens):
|
| 203 |
+
inp = ids[:, -cfg["ctx"]:].clamp(max=cfg["vocab"] - 1)
|
| 204 |
+
logits = model(inp)[:, -1, :] / max(temperature, 1e-6)
|
| 205 |
+
|
| 206 |
+
if top_k:
|
| 207 |
+
k = min(top_k, logits.size(-1))
|
| 208 |
+
vals, idx = torch.topk(logits, k)
|
| 209 |
+
probs = F.softmax(vals, dim=-1)
|
| 210 |
+
pick = torch.multinomial(probs, 1)
|
| 211 |
+
nxt = idx[0, pick].item()
|
| 212 |
+
else:
|
| 213 |
+
nxt = logits.argmax(-1)[0].item()
|
| 214 |
+
|
| 215 |
+
out.append(nxt)
|
| 216 |
+
ids = torch.cat([ids, torch.tensor([[nxt]], device=device)], dim=1)
|
| 217 |
+
|
| 218 |
+
text = tok.decode(out, skip_special_tokens=True)
|
| 219 |
+
return text
|