--- license: other language: - en tags: - text-generation - merlin - small-model - pretraining --- ![f4ea02ff-a403-4fea-8f82-66f34dc040ad](https://cdn-uploads.huggingface.co/production/uploads/6a5f8c87a38cac087c2c6c05/TgW4BJvq49nFce0U940rh.jpeg) # Merlin-9M **Merlin-9M** is an **8.85M-parameter** decoder-only language model, trained from scratch on **Stew-10B** (Wikipedia / Ultra-FineWeb / DCLM) with **no benchmark data of any kind** in the training mix. It is the checkpoint from step 12,502 of training, seen ~811M tokens. ## Architecture | Property | Value | |---|---| | Parameters | 8,854,146 | | `d_model` | 256 | | Layers | 12 unique | | Heads | 8 (head_dim 32) | | FFN | SwiGLU(384) | | Vocabulary | 8,192 (GPT-2's first 8192 merges) | | Context | 1024 | | RoPE | yes | | Normalization | RMSNorm + QK-norm | | Activation | SwiGLU | | Output | logits softcap 15 | | Optimizer | Muon (2D weights) + AdamW (embeddings) | ### Depth-recurrent core Layers 3–8 are applied **twice**, giving **18 effective layers** from 12 unique ones at zero extra parameter cost. The first pass uses a learned residual scale; the second pass uses a smaller one (`scale_loop`). ### Multi-token prediction (MTP-2) The training objective adds an auxiliary head predicting the token two positions ahead, tied to the same embedding matrix. This does not affect inference — the model is used as a standard causal LM. ## How to use ```python from huggingface_hub import hf_hub_download import torch # download checkpoint path = hf_hub_download(repo_id="Monster-Code/Merlin-9M", filename="brasa9m_last.pt", repo_type="model") # The checkpoint dict contains: # ck["model"] -> state_dict # ck["cfg"] -> architecture config # ck["step"] -> training step (12502) # See merlin9m.py in this repo for the model definition + a load_model() helper. ``` The `merlin9m.py` file in this repository is a self-contained inference module: ```python from merlin9m import load_model, generate model, cfg, tok, step = load_model("Monster-Code/Merlin-9M") print(generate(model, cfg, tok, "The capital of France is")) ``` ## Tokenizer GPT-2 BPE. The model was trained on the **first 8,192 token IDs** of GPT-2's 50,257-token vocabulary — these are the most frequent merges in English. Input token IDs outside `[0, 8191]` are clamped during inference. ## Training data [Monster-Code/Stew-10B-tokenized](https://huggingface.co/datasets/Monster-Code/Stew-10B-tokenized) — a 10B-token pretraining corpus: | Source | Share | Tokens | |---|---|---| | Wikipedia (`20231101.en`) | 25% | 2.5B | | Ultra-FineWeb (English) | 50% | 5.0B | | DCLM-baseline 1.0 | 25% | 2.5B | **No ARC, HellaSwag, PIQA, WinoGrande, or any other benchmark data was used in training.** ## License Mixed upstream licenses (CC BY-SA 3.0, GFDL, see upstream dataset cards). No additional restrictions are imposed by this repository.