---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- language-model
- forgeplex
- forgeworks
- rope
- swiglu
- gqa
- attn-output-gate
- refresh-gate
- custom_code
pipeline_tag: text-generation
---

<p align="center"><img src="assets/title.jpg" width="650"></p>

# ForgePlex-M2.2-10M

**ForgePlex-M2.2-10M** is the definitive M2: the final and most refined release of the ForgePlex M2 architecture from **ForgeWorks**. Same ~9.99M-parameter (unique, tied embeddings) decoder-only design as M2 and M2.1, pushed as far as we could take it. Packaged as a Hugging Face custom causal LM (**requires `trust_remote_code=True`**).

M2.2 improves on M2.1 on HellaSwag, ARC, PIQA and ArithMark, and the Intelligence Index climbs from 10.70 to **12.12**.

| | |
|---|---|
| Unique parameters | **9,991,938** |
| Post-training | **2B tokens** on top of M2.1 |
| License | Apache-2.0 |

## Architecture

GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style **attention output gates** on every layer and GPT-S2-style **refresh gates** on inject layers `[5, 10]` (kernel 9). **XSA is off.** Weights keep the original key layout (no Llama remapping).

| Component | Details |
|---|---|
| Position encoding | **RoPE** (theta=5,000, NeoX even/odd) |
| Normalization | **RMSNorm** (eps=1e-6) |
| Feed-forward | **SwiGLU** (intermediate 712) |
| Attention | **GQA**: 8Q / 2KV, head_dim=32 + attn output gate |
| Refresh | Layers **5, 10**, kernel **9** |
| Bias | **None** |
| Embedding | **Weight tying** |
| Depth x width | **11 layers x 256 hidden** |
| Context | **1024 tokens** |
| Vocab | **4,096** custom BPE |

## Evaluation

All scores are 0-shot `acc_norm`. HellaSwag, ARC and PIQA are run with `lm-eval`; ArithMark-3 is reported as `acc_norm`. ARC is the mean of ARC-Easy and ARC-Challenge.

| Benchmark | ForgePlex-M2.2-10M |
|---|---|
| HellaSwag | **29.19** |
| ARC-Easy | **39.77** |
| ARC-Challenge | **22.70** |
| ARC (avg) | **31.24** |
| PIQA | **59.79** |
| ArithMark-3 | **37.40** |
| **Intelligence Index** | **12.12** |

### Compared with earlier M2 releases

| Model | HellaSwag | ARC-e | ARC-c | ARC (avg) | PIQA | ArithMark-3 | Index |
|---|---|---|---|---|---|---|---|
| ForgePlex-M2 | 28.02 | - | - | 30.00 | 57.18 | 34.60 | 9.14 |
| ForgePlex-M2.1-10M | 29.09 | 36.45 | 22.87 | 29.66 | 59.14 | 35.50 | 10.70 |
| **ForgePlex-M2.2-10M** | **29.19** | **39.77** | 22.70 | **31.24** | **59.79** | **37.40** | **12.12** |

Against M2.1, M2.2 is up on HellaSwag (+0.10), ARC (+1.58 avg, driven by +3.32 on ARC-Easy, with ARC-Challenge roughly flat at -0.17), PIQA (+0.65) and ArithMark (+1.90), for +1.42 on the Index. A balanced, jack-of-all-trades step up rather than a spike on one benchmark.

### Against other <10M models

| Model | Intelligence Index | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark |
|---|---|---|---|---|---|---|
| [ForgePlex-M2.2-10M](https://huggingface.co/ForgeWorks/ForgePlex-M2.2-10M) | **12.12** | **29.19** | **39.77** | 22.70 | **59.79** | **37.40** |
| [min-spark-1.6](https://huggingface.co/MinimaLabs/min-spark-1.6) | 11.08 | 28.52 | 38.30 | 22.44 | 59.14 | 36.90 |
| [heron1i-flash-lite-v2](https://huggingface.co/Heronetic/heron1i-flash-lite-v2) | 11.11 | 28.86 | 38.89 | **23.38** | 58.54 | 36.70 |

M2.2 leads on HellaSwag, ARC-Easy, PIQA and ArithMark against the strongest sub-10M models, while staying close on ARC-Challenge.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ForgeWorks/ForgePlex-M2.2-10M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

Or run `python usage.py` from a local copy of this repo.

## Citation

```bibtex
@misc{forgeplex_m22_10m,
  title={ForgePlex-M2.2-10M},
  author={ForgeWorks},
  year={2026},
  howpublished={\url{https://huggingface.co/ForgeWorks/ForgePlex-M2.2-10M}}
}
```