---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - grt
  - geometric-rotation-transformer
  - mixture-of-rotations
  - mor
  - moe
  - rotational-transformer
  - givens-rotations
  - clifford-algebra
  - matterwave
base_model: MatterWaveAI/grt-30m
---

# GRT-30M-MoR (`MatterWaveAI/grt-30m-mor`)

**GRT-30M-MoR** (`31,634,086` parameters, **zero `nn.Linear` layers**) is the **Mixture-of-Rotations (`MoR`)** upcycled variant of [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` parameters).

Instead of routing tokens through multiple separate dense `nn.Linear` feed-forward matrices, **GRT-30M-MoR** exploits the abelian Lie group structure of planar `2×2` Givens rotations (`SO(2)^P` at each butterfly stage): the router blends the `E = 4` expert rotation angles and log-scales in parameter space **before** executing a single `cos / sin` Givens rotation cascade per stage — providing **`4×` FFN rotational capacity at `1×` trigonometric rotation compute**.

---

## Model Details

| Property | Value |
| :--- | :--- |
| **Architecture** | `GRTForCausalLM` (`model_type: "grt"`, Mixture-of-Rotations) |
| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
| **Total stored parameters** | `31,634,086` (`31.63M`, `+2,328,588` over `grt-30m`) |
| **Non-embedding backbone parameters** | `12,068,785` (`12.07M` across 12 MoR blocks) |
| **Number of FFN rotational experts (`ffn_n_experts`)** | `4` per block (`48` rotational FFN experts across 12 layers) |
| **MoR parameterization (`mor_mode`)** | `"direct"` (`expert_expand_angles [4, 21, 512]`, `expert_expand_scales [4, 21, 1024]`, `expert_contract_angles [4, 21, 512]`, `expert_contract_scales [4, 21, 1024]`) |
| **Router architecture (`mor_router_mode`)** | `"rotational"` (`GRTRotationalRouter` — cosine phase-angle alignment on `(S¹)^128`, **0 `nn.Linear` layers**) |
| **`nn.Linear` layers** | **`0`** |
| **`hidden_size` (`d_model`) / `intermediate_size` (`d_ff`)** | `256` / `1,024` |
| **`num_hidden_layers` / `num_attention_heads`** | `12` / `4` (`head_dim = 64`) |
| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
| **Vocabulary** | `151,669` (Qwen 3 BPE, tied polar cosine head) |

---

## Architectural Highlights

1. **4× FFN Rotational Capacity at 1× Rotation Compute (`_mor_rotate`)**:
   Each of the 12 `GRTFeedForward` blocks maintains `ffn_n_experts = 4` rotational cascades. Given per-token router probabilities `w[n, e]` (`sum_e w[n, e] = 1`), the effective rotation angle and positive scale at stage `k`, pair `p` / dimension `d` are blended prior to rotation:
   ```text
   θ_blended[n, k, p] = Σ_{e=0..3} w[n, e] · θ_expert[e, k, p]
   s_blended[n, k, d] = Σ_{e=0..3} w[n, e] · softplus(s_raw_expert[e, k, d])
   ```
   A single Givens rotation `(cos θ_blended, sin θ_blended)` is then applied per pair.
2. **100% Rotational Router (`GRTRotationalRouter`, 0 `nn.Linear` Layers)**:
   Tokens are routed to experts via scaled cosine alignment between L2-normalized planar coordinate pairs `(u_p, v_p)` and `E = 4` learned phase-angle prototypes `(cos φ_{e,p}, sin φ_{e,p})` with learned `softplus(output_scale)` temperature.
3. **Zero-Sum Walsh–Hadamard Upcycling & Multi-Domain Specialization (`PairAdam`)**:
   Initialized from `MatterWaveAI/grt-30m` via zero-sum Walsh–Hadamard angular perturbations (`Σ_e Δ^(e) = 0`) and specialized on a 4-domain corpus (`rotational_geometry`, `science_and_physics`, `code_and_algorithms`, `reasoning_and_language`) on an Intel Arc 140T XPU (`xpu:0`) with `PairAdam` and Switch-style load balancing:

| Metric | Initial Upcycle | After MoR Specialization (`PairAdam`, XPU) | After MoR Specialization (`PairAdam`, ROCm `gfx1151`) |
| :--- | :---: | :---: | :---: |
| **Held-Out Validation Loss** | `5.8467` | `0.5115` | **`0.4829`** |
| **Held-Out Validation Perplexity** | `346.10` | `1.67` | **`1.62`** |
| **Domain Loss — `code_and_algorithms`** | `5.4092` | `0.4334` | **`0.3669`** |
| **Domain Loss — `rotational_geometry`** | `6.6421` | `0.4991` | **`0.4772`** (`0.4747` w/ `PolarNaturalAdam`) |
| **Domain Loss — `science_and_physics`** | `5.3337` | `0.5293` | **`0.5155`** |
| **Domain Loss — `reasoning_and_language`** | `6.1720` | `0.6442` | **`0.6466`** (w/ `PolarNaturalAdam`) |
| **Mean Expert Routing Load (`E = 4`)** | `[0.25, 0.25, 0.25, 0.25]` | `[0.250, 0.248, 0.256, 0.245]` | **`[0.250, 0.250, 0.254, 0.246]`** (`entropy = 1.9999 / 2.0000 bits`) |

---

## 8-Way Classical Geometric Optimizer Ablation (`Spark ROCm gfx1151`)

We benchmarked 8 classical geometric, symplectic, and gyroscopic optimizers from `transformers.models.grt.optimization_grt` on `MatterWaveAI/grt-30m-mor` (`80` Phase-2 specialization steps on `blocks[8:12]` + `final_norm` from an identical Phase-1 cached prefix state, `batch_size = 36`, `lr = 3.5e-3`, `max_angular_step = 0.08`, `seed = 42`):

| Rank | Optimizer (`optimizer_type`) | Held-Out Val Loss | Held-Out Val PPL | `code_and_algorithms` Loss | `rotational_geometry` Loss | `science_and_physics` Loss | `reasoning_and_language` Loss | Routing Entropy (bits / `2.0000`) | Top-4 Expert Angle L2 Div | All-Layer Angle L2 Div | Time (s) |
| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **1** | **`PairAdam` (`pair_adam`)** | **`0.4829`** | **`1.6208`** | **`0.3669`** | `0.4772` | **`0.5155`** | `0.6553` | `1.9999` (`1.3862` nats) | `2.5347` | `2.5386` | `180.15s` |
| **2** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `0.4848` | `1.6239` | `0.3744` | **`0.4747`** | `0.5226` | **`0.6466`** | `1.9999` (`1.3862` nats) | `2.5349` | `2.5387` | `180.75s` |
| **3** | **`CyclideAdam` (`cyclide_adam`)** | `0.5006` | `1.6498` | `0.3758` | `0.4986` | `0.5426` | `0.6718` | `1.9999` (`1.3862` nats) | `2.5789` | `2.5534` | `180.09s` |
| **4** | **`HuygensAdam` (`huygens_adam`)** | `0.5225` | `1.6862` | `0.3957` | `0.5107` | `0.5696` | `0.7041` | `1.9999` (`1.3862` nats) | **`2.6216`** | **`2.5676`** | `176.07s` |
| **5** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | `0.5230` | `1.6871` | `0.4091` | `0.5202` | `0.5570` | `0.6859` | `1.9999` (`1.3862` nats) | `2.5351` | `2.5388` | **`176.01s`** |
| **6** | **`HessianPairAdam` (`hessian_pair_adam`)** | `0.5403` | `1.7165` | `0.4082` | `0.5330` | `0.5916` | `0.7201` | `1.9999` (`1.3862` nats) | `2.5349` | `2.5387` | `180.09s` |
| **7** | **`SymplecticAdam` (`symplectic_adam`)** | `0.5567` | `1.7448` | `0.4185` | `0.5530` | `0.6082` | `0.7417` | `1.9999` (`1.3862` nats) | `2.5345` | `2.5386` | `180.20s` |
| **8** | **`GyricAdam` (`gyric_adam`)** | `0.5745` | `1.7763` | `0.4279` | `0.5672` | `0.6412` | `0.7609` | `1.9999` (`1.3862` nats) | `2.5350` | `2.5387` | `177.29s` |

### Key Geometric Findings on Mixture-of-Rotations (`ffn_n_experts = 4`)
- **Huygens Circular Order Parameter Splay Torque Prevents Expert Collapse**: On 3D MoR expert tensors (`expert_expand_angles [4, 21, 512]` and `expert_contract_angles [4, 21, 512]`), `HuygensAdam` computes the circular synchronization order parameter `R_p = |(1/4) ∑_{e=0}^3 exp(i θ_{e,k,p})|` directly across dimension `0` (`E = 4` experts) and applies a zero-mean splay restoring torque whenever experts over-synchronize, achieving the **highest pairwise expert angle L2 divergence (`2.6216` in top-4 MoR blocks, `2.5676` all-layer mean)** while maintaining `1.9999 / 2.0000` bits of routing entropy.
- **Polar Riemannian Metric Excels on Conceptual & Geometric Domains**: `PolarNaturalAdam` achieves **#1 validation loss on `rotational_geometry` (`0.4747`)** and **#1 validation loss on `reasoning_and_language` (`0.6466`)**, finishing within `0.0019` overall loss of `PairAdam` (`0.4848` vs. `0.4829`).
- **Curriculum Advantage of `HuygensSymplecticAdam`**: Combining early-stage Huygens expert splay repulsion with late-stage Gromov–Liouville symplectic non-squeezing (`HuygensSymplecticAdam`, `0.5230` loss / `1.6871` PPL) outperforms standalone `SymplecticAdam` (`0.5567`), `HessianPairAdam` (`0.5403`), and `GyricAdam` (`0.5745`).

---

## Quickstart (`transformers`)

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-mor"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)

inputs = tokenizer("The Givens Rotational Transformer replaces dense matrices with", return_tensors="pt")
out_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out_ids[0], skip_special_tokens=True))
```

## Quickstart (`vLLM`)

```python
from vllm import LLM, SamplingParams

llm = LLM(model="MatterWaveAI/grt-30m-mor", model_impl="transformers", dtype="float32")
outputs = llm.generate(
    ["The Givens Rotational Transformer replaces dense matrices with"],
    SamplingParams(temperature=0.0, max_tokens=32),
)
print(outputs[0].outputs[0].text)
```

---

## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face

1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)

- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)

---

## License

Apache 2.0

---

## Citation

```bibtex
@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}
```