---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- language-model
- forgeplex
- forgeworks
- rope
- swiglu
- gqa
- attn-output-gate
- refresh-gate
- custom_code
pipeline_tag: text-generation
---

<p align="center"><img src="assets/title.jpg" width="650"></p>

# ForgePlex-M2.1-10M

**ForgePlex-M2.1-10M** is a 9.99M-parameter decoder-only language model from ForgeWorks, trained on 40b tokens. This is the most complete model we've built so far. We thank the ForgeWorks team alongside anyone who delivered assistance in the creation of this model.

# Why not name it ForgePlex-M3? 

We decided to give the model the M2.1 classification because at its core it's still M2. The changes are refinements: a slightly wider feed-forward layer (707 to 712), a retrained tokenizer, and a longer, better-tuned training run. A new major version should mean a real architectural step, so we're saving the M3 name for that.

| Metric | Value |
|---|---|
| Unique parameters | **9,991,938** |
| Intelligence Index |**10.70** |
| HellaSwag | 	29.09% |
| ARC (combined) | 	29.66% |
| PIQA | 	59.14% |
| ArithMark-3 | 	35.50% |
| License | Apache-2.0 |

## Architecture

GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style **attention output gates** on every layer and GPT-S2-style **refresh gates** on inject layers `[5, 10]` (kernel 9). **XSA is off.** Weights keep the original key layout (no Llama remapping).

| Component | Details |
|---|---|
| Position encoding | **RoPE** (theta=5,000, NeoX even/odd) |
| Normalization | **RMSNorm** (eps=1e-6) |
| Feed-forward | **SwiGLU** (intermediate 712) |
| Attention | **GQA**: 8Q / 2KV, head_dim=32 + attn output gate |
| Refresh | Layers **5, 10**, kernel **9** |
| Bias | **None** |
| Embedding | **Weight tying** |
| Depth x width | **11 layers x 256 hidden** |
| Context | **1024 tokens** |
| Vocab | **4,096** custom BPE |

## ForgePlex-M2.1-10M vs <10m Leaders

<img src="assets/chart_competitors.png" width="550">
ForgePlex-M2.1 leads the current <10M top models on HellaSwag and PIQA, and stays competitive on ARC and ArithMark.
<img src="assets/chart_m2_vs_m21.png" width="550">
M2.1 improves on M2 in HellaSwag, PIQA and ArithMark, and holds ARC roughly level.
## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ForgeWorks/ForgePlex-M2.1-10M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

Or run `python usage.py` from a local copy of this repo.

```bibtex
@misc{forgeplex_m21_10m,
  title={ForgePlex-M2.1-10M},
  author={ForgeWorks},
  year={2026},
  howpublished={\url{https://huggingface.co/ForgeWorks/ForgePlex-M2.1-10M}}
}
```