Instructions to use Monster-Code/Merlin-9M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Merlin
How to use Monster-Code/Merlin-9M with Merlin:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from Monster-Code/Merlin-9M: direct link, hf CLI and curl.
- Browser
- Download file 2.93 kB
-
https://huggingface.co/Monster-Code/Merlin-9M/resolve/main/README.md
- Command line
-
hf download hf://Monster-Code/Merlin-9M/README.md
-
curl -L -o README.md https://huggingface.co/Monster-Code/Merlin-9M/resolve/main/README.md
2.93 kB
| license: other | |
| language: | |
| - en | |
| tags: | |
| - text-generation | |
| - merlin | |
| - small-model | |
| - pretraining | |
|  | |
| # Merlin-9M | |
| **Merlin-9M** is an **8.85M-parameter** decoder-only language model, trained | |
| from scratch on **Stew-10B** (Wikipedia / Ultra-FineWeb / DCLM) with **no | |
| benchmark data of any kind** in the training mix. It is the checkpoint from | |
| step 12,502 of training, seen ~811M tokens. | |
| ## Architecture | |
| | Property | Value | | |
| |---|---| | |
| | Parameters | 8,854,146 | | |
| | `d_model` | 256 | | |
| | Layers | 12 unique | | |
| | Heads | 8 (head_dim 32) | | |
| | FFN | SwiGLU(384) | | |
| | Vocabulary | 8,192 (GPT-2's first 8192 merges) | | |
| | Context | 1024 | | |
| | RoPE | yes | | |
| | Normalization | RMSNorm + QK-norm | | |
| | Activation | SwiGLU | | |
| | Output | logits softcap 15 | | |
| | Optimizer | Muon (2D weights) + AdamW (embeddings) | | |
| ### Depth-recurrent core | |
| Layers 3–8 are applied **twice**, giving **18 effective layers** from 12 unique | |
| ones at zero extra parameter cost. The first pass uses a learned residual scale; | |
| the second pass uses a smaller one (`scale_loop`). | |
| ### Multi-token prediction (MTP-2) | |
| The training objective adds an auxiliary head predicting the token two positions | |
| ahead, tied to the same embedding matrix. This does not affect inference — the | |
| model is used as a standard causal LM. | |
| ## How to use | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| import torch | |
| # download checkpoint | |
| path = hf_hub_download(repo_id="Monster-Code/Merlin-9M", | |
| filename="brasa9m_last.pt", repo_type="model") | |
| # The checkpoint dict contains: | |
| # ck["model"] -> state_dict | |
| # ck["cfg"] -> architecture config | |
| # ck["step"] -> training step (12502) | |
| # See merlin9m.py in this repo for the model definition + a load_model() helper. | |
| ``` | |
| The `merlin9m.py` file in this repository is a self-contained inference module: | |
| ```python | |
| from merlin9m import load_model, generate | |
| model, cfg, tok, step = load_model("Monster-Code/Merlin-9M") | |
| print(generate(model, cfg, tok, "The capital of France is")) | |
| ``` | |
| ## Tokenizer | |
| GPT-2 BPE. The model was trained on the **first 8,192 token IDs** of GPT-2's | |
| 50,257-token vocabulary — these are the most frequent merges in English. Input | |
| token IDs outside `[0, 8191]` are clamped during inference. | |
| ## Training data | |
| [Monster-Code/Stew-10B-tokenized](https://huggingface.co/datasets/Monster-Code/Stew-10B-tokenized) | |
| — a 10B-token pretraining corpus: | |
| | Source | Share | Tokens | | |
| |---|---|---| | |
| | Wikipedia (`20231101.en`) | 25% | 2.5B | | |
| | Ultra-FineWeb (English) | 50% | 5.0B | | |
| | DCLM-baseline 1.0 | 25% | 2.5B | | |
| **No ARC, HellaSwag, PIQA, WinoGrande, or any other benchmark data was used | |
| in training.** | |
| ## License | |
| Mixed upstream licenses (CC BY-SA 3.0, GFDL, see upstream dataset cards). | |
| No additional restrictions are imposed by this repository. |