Merlin-9M / README.md
Monster-Code's picture
Update README.md
22f991b verified
|
Raw History Blame
2.93 kB
---
license: other
language:
- en
tags:
- text-generation
- merlin
- small-model
- pretraining
---
![f4ea02ff-a403-4fea-8f82-66f34dc040ad](https://cdn-uploads.huggingface.co/production/uploads/6a5f8c87a38cac087c2c6c05/TgW4BJvq49nFce0U940rh.jpeg)
# Merlin-9M
**Merlin-9M** is an **8.85M-parameter** decoder-only language model, trained
from scratch on **Stew-10B** (Wikipedia / Ultra-FineWeb / DCLM) with **no
benchmark data of any kind** in the training mix. It is the checkpoint from
step 12,502 of training, seen ~811M tokens.
## Architecture
| Property | Value |
|---|---|
| Parameters | 8,854,146 |
| `d_model` | 256 |
| Layers | 12 unique |
| Heads | 8 (head_dim 32) |
| FFN | SwiGLU(384) |
| Vocabulary | 8,192 (GPT-2's first 8192 merges) |
| Context | 1024 |
| RoPE | yes |
| Normalization | RMSNorm + QK-norm |
| Activation | SwiGLU |
| Output | logits softcap 15 |
| Optimizer | Muon (2D weights) + AdamW (embeddings) |
### Depth-recurrent core
Layers 3–8 are applied **twice**, giving **18 effective layers** from 12 unique
ones at zero extra parameter cost. The first pass uses a learned residual scale;
the second pass uses a smaller one (`scale_loop`).
### Multi-token prediction (MTP-2)
The training objective adds an auxiliary head predicting the token two positions
ahead, tied to the same embedding matrix. This does not affect inference — the
model is used as a standard causal LM.
## How to use
```python
from huggingface_hub import hf_hub_download
import torch
# download checkpoint
path = hf_hub_download(repo_id="Monster-Code/Merlin-9M",
filename="brasa9m_last.pt", repo_type="model")
# The checkpoint dict contains:
# ck["model"] -> state_dict
# ck["cfg"] -> architecture config
# ck["step"] -> training step (12502)
# See merlin9m.py in this repo for the model definition + a load_model() helper.
```
The `merlin9m.py` file in this repository is a self-contained inference module:
```python
from merlin9m import load_model, generate
model, cfg, tok, step = load_model("Monster-Code/Merlin-9M")
print(generate(model, cfg, tok, "The capital of France is"))
```
## Tokenizer
GPT-2 BPE. The model was trained on the **first 8,192 token IDs** of GPT-2's
50,257-token vocabulary — these are the most frequent merges in English. Input
token IDs outside `[0, 8191]` are clamped during inference.
## Training data
[Monster-Code/Stew-10B-tokenized](https://huggingface.co/datasets/Monster-Code/Stew-10B-tokenized)
— a 10B-token pretraining corpus:
| Source | Share | Tokens |
|---|---|---|
| Wikipedia (`20231101.en`) | 25% | 2.5B |
| Ultra-FineWeb (English) | 50% | 5.0B |
| DCLM-baseline 1.0 | 25% | 2.5B |
**No ARC, HellaSwag, PIQA, WinoGrande, or any other benchmark data was used
in training.**
## License
Mixed upstream licenses (CC BY-SA 3.0, GFDL, see upstream dataset cards).
No additional restrictions are imposed by this repository.