| ---
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - grt
|
| - geometric-rotation-transformer
|
| - mixture-of-rotations
|
| - mor
|
| - moe
|
| - rotational-transformer
|
| - givens-rotations
|
| - clifford-algebra
|
| - matterwave
|
| base_model: MatterWaveAI/grt-30m
|
| ---
|
|
|
| # GRT-30M-MoR (`MatterWaveAI/grt-30m-mor`)
|
|
|
| **GRT-30M-MoR** (`31,634,086` parameters, **zero `nn.Linear` layers**) is the **Mixture-of-Rotations (`MoR`)** upcycled variant of [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` parameters).
|
|
|
| Instead of routing tokens through multiple separate dense `nn.Linear` feed-forward matrices, **GRT-30M-MoR** exploits the abelian Lie group structure of planar `2×2` Givens rotations (`SO(2)^P` at each butterfly stage): the router blends the `E = 4` expert rotation angles and log-scales in parameter space **before** executing a single `cos / sin` Givens rotation cascade per stage — providing **`4×` FFN rotational capacity at `1×` trigonometric rotation compute**.
|
|
|
| ---
|
|
|
| ## Model Details
|
|
|
| | Property | Value |
|
| | :--- | :--- |
|
| | **Architecture** | `GRTForCausalLM` (`model_type: "grt"`, Mixture-of-Rotations) |
|
| | **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
|
| | **Total stored parameters** | `31,634,086` (`31.63M`, `+2,328,588` over `grt-30m`) |
|
| | **Non-embedding backbone parameters** | `12,068,785` (`12.07M` across 12 MoR blocks) |
|
| | **Number of FFN rotational experts (`ffn_n_experts`)** | `4` per block (`48` rotational FFN experts across 12 layers) |
|
| | **MoR parameterization (`mor_mode`)** | `"direct"` (`expert_expand_angles [4, 21, 512]`, `expert_expand_scales [4, 21, 1024]`, `expert_contract_angles [4, 21, 512]`, `expert_contract_scales [4, 21, 1024]`) |
|
| | **Router architecture (`mor_router_mode`)** | `"rotational"` (`GRTRotationalRouter` — cosine phase-angle alignment on `(S¹)^128`, **0 `nn.Linear` layers**) |
|
| | **`nn.Linear` layers** | **`0`** |
|
| | **`hidden_size` (`d_model`) / `intermediate_size` (`d_ff`)** | `256` / `1,024` |
|
| | **`num_hidden_layers` / `num_attention_heads`** | `12` / `4` (`head_dim = 64`) |
|
| | **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
|
| | **Vocabulary** | `151,669` (Qwen 3 BPE, tied polar cosine head) |
|
|
|
| ---
|
|
|
| ## Architectural Highlights
|
|
|
| 1. **4× FFN Rotational Capacity at 1× Rotation Compute (`_mor_rotate`)**:
|
| Each of the 12 `GRTFeedForward` blocks maintains `ffn_n_experts = 4` rotational cascades. Given per-token router probabilities `w[n, e]` (`sum_e w[n, e] = 1`), the effective rotation angle and positive scale at stage `k`, pair `p` / dimension `d` are blended prior to rotation:
|
| ```text
|
| θ_blended[n, k, p] = Σ_{e=0..3} w[n, e] · θ_expert[e, k, p]
|
| s_blended[n, k, d] = Σ_{e=0..3} w[n, e] · softplus(s_raw_expert[e, k, d])
|
| ```
|
| A single Givens rotation `(cos θ_blended, sin θ_blended)` is then applied per pair.
|
| 2. **100% Rotational Router (`GRTRotationalRouter`, 0 `nn.Linear` Layers)**:
|
| Tokens are routed to experts via scaled cosine alignment between L2-normalized planar coordinate pairs `(u_p, v_p)` and `E = 4` learned phase-angle prototypes `(cos φ_{e,p}, sin φ_{e,p})` with learned `softplus(output_scale)` temperature.
|
| 3. **Zero-Sum Walsh–Hadamard Upcycling & Multi-Domain Specialization (`PairAdam`)**:
|
| Initialized from `MatterWaveAI/grt-30m` via zero-sum Walsh–Hadamard angular perturbations (`Σ_e Δ^(e) = 0`) and specialized on a 4-domain corpus (`rotational_geometry`, `science_and_physics`, `code_and_algorithms`, `reasoning_and_language`) on an Intel Arc 140T XPU (`xpu:0`) with `PairAdam` and Switch-style load balancing:
|
|
|
| | Metric | Initial Upcycle | After MoR Specialization (`PairAdam`, XPU) | After MoR Specialization (`PairAdam`, ROCm `gfx1151`) |
|
| | :--- | :---: | :---: | :---: |
|
| | **Held-Out Validation Loss** | `5.8467` | `0.5115` | **`0.4829`** |
|
| | **Held-Out Validation Perplexity** | `346.10` | `1.67` | **`1.62`** |
|
| | **Domain Loss — `code_and_algorithms`** | `5.4092` | `0.4334` | **`0.3669`** |
|
| | **Domain Loss — `rotational_geometry`** | `6.6421` | `0.4991` | **`0.4772`** (`0.4747` w/ `PolarNaturalAdam`) |
|
| | **Domain Loss — `science_and_physics`** | `5.3337` | `0.5293` | **`0.5155`** |
|
| | **Domain Loss — `reasoning_and_language`** | `6.1720` | `0.6442` | **`0.6466`** (w/ `PolarNaturalAdam`) |
|
| | **Mean Expert Routing Load (`E = 4`)** | `[0.25, 0.25, 0.25, 0.25]` | `[0.250, 0.248, 0.256, 0.245]` | **`[0.250, 0.250, 0.254, 0.246]`** (`entropy = 1.9999 / 2.0000 bits`) |
|
|
|
| ---
|
|
|
| ## 8-Way Classical Geometric Optimizer Ablation (`Spark ROCm gfx1151`)
|
|
|
| We benchmarked 8 classical geometric, symplectic, and gyroscopic optimizers from `transformers.models.grt.optimization_grt` on `MatterWaveAI/grt-30m-mor` (`80` Phase-2 specialization steps on `blocks[8:12]` + `final_norm` from an identical Phase-1 cached prefix state, `batch_size = 36`, `lr = 3.5e-3`, `max_angular_step = 0.08`, `seed = 42`):
|
|
|
| | Rank | Optimizer (`optimizer_type`) | Held-Out Val Loss | Held-Out Val PPL | `code_and_algorithms` Loss | `rotational_geometry` Loss | `science_and_physics` Loss | `reasoning_and_language` Loss | Routing Entropy (bits / `2.0000`) | Top-4 Expert Angle L2 Div | All-Layer Angle L2 Div | Time (s) |
|
| | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| | **1** | **`PairAdam` (`pair_adam`)** | **`0.4829`** | **`1.6208`** | **`0.3669`** | `0.4772` | **`0.5155`** | `0.6553` | `1.9999` (`1.3862` nats) | `2.5347` | `2.5386` | `180.15s` |
|
| | **2** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `0.4848` | `1.6239` | `0.3744` | **`0.4747`** | `0.5226` | **`0.6466`** | `1.9999` (`1.3862` nats) | `2.5349` | `2.5387` | `180.75s` |
|
| | **3** | **`CyclideAdam` (`cyclide_adam`)** | `0.5006` | `1.6498` | `0.3758` | `0.4986` | `0.5426` | `0.6718` | `1.9999` (`1.3862` nats) | `2.5789` | `2.5534` | `180.09s` |
|
| | **4** | **`HuygensAdam` (`huygens_adam`)** | `0.5225` | `1.6862` | `0.3957` | `0.5107` | `0.5696` | `0.7041` | `1.9999` (`1.3862` nats) | **`2.6216`** | **`2.5676`** | `176.07s` |
|
| | **5** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | `0.5230` | `1.6871` | `0.4091` | `0.5202` | `0.5570` | `0.6859` | `1.9999` (`1.3862` nats) | `2.5351` | `2.5388` | **`176.01s`** |
|
| | **6** | **`HessianPairAdam` (`hessian_pair_adam`)** | `0.5403` | `1.7165` | `0.4082` | `0.5330` | `0.5916` | `0.7201` | `1.9999` (`1.3862` nats) | `2.5349` | `2.5387` | `180.09s` |
|
| | **7** | **`SymplecticAdam` (`symplectic_adam`)** | `0.5567` | `1.7448` | `0.4185` | `0.5530` | `0.6082` | `0.7417` | `1.9999` (`1.3862` nats) | `2.5345` | `2.5386` | `180.20s` |
|
| | **8** | **`GyricAdam` (`gyric_adam`)** | `0.5745` | `1.7763` | `0.4279` | `0.5672` | `0.6412` | `0.7609` | `1.9999` (`1.3862` nats) | `2.5350` | `2.5387` | `177.29s` |
|
|
|
| ### Key Geometric Findings on Mixture-of-Rotations (`ffn_n_experts = 4`)
|
| - **Huygens Circular Order Parameter Splay Torque Prevents Expert Collapse**: On 3D MoR expert tensors (`expert_expand_angles [4, 21, 512]` and `expert_contract_angles [4, 21, 512]`), `HuygensAdam` computes the circular synchronization order parameter `R_p = |(1/4) ∑_{e=0}^3 exp(i θ_{e,k,p})|` directly across dimension `0` (`E = 4` experts) and applies a zero-mean splay restoring torque whenever experts over-synchronize, achieving the **highest pairwise expert angle L2 divergence (`2.6216` in top-4 MoR blocks, `2.5676` all-layer mean)** while maintaining `1.9999 / 2.0000` bits of routing entropy.
|
| - **Polar Riemannian Metric Excels on Conceptual & Geometric Domains**: `PolarNaturalAdam` achieves **#1 validation loss on `rotational_geometry` (`0.4747`)** and **#1 validation loss on `reasoning_and_language` (`0.6466`)**, finishing within `0.0019` overall loss of `PairAdam` (`0.4848` vs. `0.4829`).
|
| - **Curriculum Advantage of `HuygensSymplecticAdam`**: Combining early-stage Huygens expert splay repulsion with late-stage Gromov–Liouville symplectic non-squeezing (`HuygensSymplecticAdam`, `0.5230` loss / `1.6871` PPL) outperforms standalone `SymplecticAdam` (`0.5567`), `HessianPairAdam` (`0.5403`), and `GyricAdam` (`0.5745`).
|
|
|
| ---
|
|
|
| ## Quickstart (`transformers`)
|
|
|
| ```python
|
| import torch
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| model_id = "MatterWaveAI/grt-30m-mor"
|
| tokenizer = AutoTokenizer.from_pretrained(model_id)
|
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)
|
|
|
| inputs = tokenizer("The Givens Rotational Transformer replaces dense matrices with", return_tensors="pt")
|
| out_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
|
| print(tokenizer.decode(out_ids[0], skip_special_tokens=True))
|
| ```
|
|
|
| ## Quickstart (`vLLM`)
|
|
|
| ```python
|
| from vllm import LLM, SamplingParams
|
|
|
| llm = LLM(model="MatterWaveAI/grt-30m-mor", model_impl="transformers", dtype="float32")
|
| outputs = llm.generate(
|
| ["The Givens Rotational Transformer replaces dense matrices with"],
|
| SamplingParams(temperature=0.0, max_tokens=32),
|
| )
|
| print(outputs[0].outputs[0].text)
|
| ```
|
|
|
| ---
|
|
|
| ## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
|
|
|
| 1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
|
| 2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
|
| 3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
|
| 4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
|
| 5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
|
| 6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
|
| 7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
|
| 8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
|
| 9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)
|
|
|
| - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
|
|
|
| ---
|
|
|
| ## License
|
|
|
| Apache 2.0
|
|
|
| ---
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @techreport{householter2026grt,
|
| title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
|
| author = {Householter, David},
|
| institution = {MatterWave AI},
|
| year = {2026},
|
| url = {https://matterwave.ai}
|
| }
|
| ```
|
| |
