| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| tags: |
| - language-model |
| - forgeplex |
| - forgeworks |
| - rope |
| - swiglu |
| - gqa |
| - attn-output-gate |
| - refresh-gate |
| - custom_code |
| pipeline_tag: text-generation |
| --- |
| |
| <p align="center"><img src="assets/title.jpg" width="650"></p> |
|
|
| # ForgePlex-M2.2-10M |
|
|
| **ForgePlex-M2.2-10M** is the definitive M2: the final and most refined release of the ForgePlex M2 architecture from **ForgeWorks**. Same ~9.99M-parameter (unique, tied embeddings) decoder-only design as M2 and M2.1, pushed as far as we could take it. Packaged as a Hugging Face custom causal LM (**requires `trust_remote_code=True`**). |
|
|
| M2.2 improves on M2.1 on HellaSwag, ARC, PIQA and ArithMark, and the Intelligence Index climbs from 10.70 to **12.12**. |
|
|
| | | | |
| |---|---| |
| | Unique parameters | **9,991,938** | |
| | Post-training | **2B tokens** on top of M2.1 | |
| | License | Apache-2.0 | |
|
|
| ## Architecture |
|
|
| GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style **attention output gates** on every layer and GPT-S2-style **refresh gates** on inject layers `[5, 10]` (kernel 9). **XSA is off.** Weights keep the original key layout (no Llama remapping). |
|
|
| | Component | Details | |
| |---|---| |
| | Position encoding | **RoPE** (theta=5,000, NeoX even/odd) | |
| | Normalization | **RMSNorm** (eps=1e-6) | |
| | Feed-forward | **SwiGLU** (intermediate 712) | |
| | Attention | **GQA**: 8Q / 2KV, head_dim=32 + attn output gate | |
| | Refresh | Layers **5, 10**, kernel **9** | |
| | Bias | **None** | |
| | Embedding | **Weight tying** | |
| | Depth x width | **11 layers x 256 hidden** | |
| | Context | **1024 tokens** | |
| | Vocab | **4,096** custom BPE | |
| |
| ## Evaluation |
| |
| All scores are 0-shot `acc_norm`. HellaSwag, ARC and PIQA are run with `lm-eval`; ArithMark-3 is reported as `acc_norm`. ARC is the mean of ARC-Easy and ARC-Challenge. |
| |
| | Benchmark | ForgePlex-M2.2-10M | |
| |---|---| |
| | HellaSwag | **29.19** | |
| | ARC-Easy | **39.77** | |
| | ARC-Challenge | **22.70** | |
| | ARC (avg) | **31.24** | |
| | PIQA | **59.79** | |
| | ArithMark-3 | **37.40** | |
| | **Intelligence Index** | **12.12** | |
| |
| ### Compared with earlier M2 releases |
| |
| | Model | HellaSwag | ARC-e | ARC-c | ARC (avg) | PIQA | ArithMark-3 | Index | |
| |---|---|---|---|---|---|---|---| |
| | ForgePlex-M2 | 28.02 | - | - | 30.00 | 57.18 | 34.60 | 9.14 | |
| | ForgePlex-M2.1-10M | 29.09 | 36.45 | 22.87 | 29.66 | 59.14 | 35.50 | 10.70 | |
| | **ForgePlex-M2.2-10M** | **29.19** | **39.77** | 22.70 | **31.24** | **59.79** | **37.40** | **12.12** | |
| |
| Against M2.1, M2.2 is up on HellaSwag (+0.10), ARC (+1.58 avg, driven by +3.32 on ARC-Easy, with ARC-Challenge roughly flat at -0.17), PIQA (+0.65) and ArithMark (+1.90), for +1.42 on the Index. A balanced, jack-of-all-trades step up rather than a spike on one benchmark. |
| |
| ### Against other <10M models |
| |
| | Model | Intelligence Index | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark | |
| |---|---|---|---|---|---|---| |
| | [ForgePlex-M2.2-10M](https://huggingface.co/ForgeWorks/ForgePlex-M2.2-10M) | **12.12** | **29.19** | **39.77** | 22.70 | **59.79** | **37.40** | |
| | [min-spark-1.6](https://huggingface.co/MinimaLabs/min-spark-1.6) | 11.08 | 28.52 | 38.30 | 22.44 | 59.14 | 36.90 | |
| | [heron1i-flash-lite-v2](https://huggingface.co/Heronetic/heron1i-flash-lite-v2) | 11.11 | 28.86 | 38.89 | **23.38** | 58.54 | 36.70 | |
| |
| M2.2 leads on HellaSwag, ARC-Easy, PIQA and ArithMark against the strongest sub-10M models, while staying close on ARC-Challenge. |
| |
| ## Usage |
| |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "ForgeWorks/ForgePlex-M2.2-10M" |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| torch_dtype=torch.float32, |
| device_map="auto", |
| ) |
| |
| prompt = "Once upon a time" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| with torch.inference_mode(): |
| out = model.generate(**inputs, max_new_tokens=80, do_sample=False) |
| print(tokenizer.decode(out[0], skip_special_tokens=True)) |
| ``` |
| |
| Or run `python usage.py` from a local copy of this repo. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{forgeplex_m22_10m, |
| title={ForgePlex-M2.2-10M}, |
| author={ForgeWorks}, |
| year={2026}, |
| howpublished={\url{https://huggingface.co/ForgeWorks/ForgePlex-M2.2-10M}} |
| } |
| ``` |
|
|
