GRT-30M-MoR (MatterWaveAI/grt-30m-mor)

GRT-30M-MoR (31,634,086 parameters, zero nn.Linear layers) is the Mixture-of-Rotations (MoR) upcycled variant of MatterWaveAI/grt-30m (29,305,498 parameters).

Instead of routing tokens through multiple separate dense nn.Linear feed-forward matrices, GRT-30M-MoR exploits the abelian Lie group structure of planar 2×2 Givens rotations (SO(2)^P at each butterfly stage): the router blends the E = 4 expert rotation angles and log-scales in parameter space before executing a single cos / sin Givens rotation cascade per stage — providing 4× FFN rotational capacity at 1× trigonometric rotation compute.

---

Model Details

Property	Value
Architecture	GRTForCausalLM (model_type: "grt", Mixture-of-Rotations)
Base model	MatterWaveAI/grt-30m
Total stored parameters	31,634,086 (31.63M, +2,328,588 over grt-30m)
Non-embedding backbone parameters	12,068,785 (12.07M across 12 MoR blocks)
Number of FFN rotational experts (ffn_n_experts)	4 per block (48 rotational FFN experts across 12 layers)
MoR parameterization (mor_mode)	"direct" (expert_expand_angles [4, 21, 512], expert_expand_scales [4, 21, 1024], expert_contract_angles [4, 21, 512], expert_contract_scales [4, 21, 1024])
Router architecture (mor_router_mode)	"rotational" (GRTRotationalRouter — cosine phase-angle alignment on (S¹)^128, 0 nn.Linear layers)
nn.Linear layers	0
hidden_size (d_model) / intermediate_size (d_ff)	256 / 1,024
num_hidden_layers / num_attention_heads	12 / 4 (head_dim = 64)
n_qkv_rotations / n_mixer_rotations / ffn_n_rotations	11 / 1 / 21
Vocabulary	151,669 (Qwen 3 BPE, tied polar cosine head)

---

Architectural Highlights

1. 4× FFN Rotational Capacity at 1× Rotation Compute (_mor_rotate): Each of the 12 GRTFeedForward blocks maintains ffn_n_experts = 4 rotational cascades. Given per-token router probabilities w[n, e] (sum_e w[n, e] = 1), the effective rotation angle and positive scale at stage k, pair p / dimension d are blended prior to rotation:

   θ_blended[n, k, p] = Σ_{e=0..3} w[n, e] · θ_expert[e, k, p]
   s_blended[n, k, d] = Σ_{e=0..3} w[n, e] · softplus(s_raw_expert[e, k, d])
   

   A single Givens rotation (cos θ_blended, sin θ_blended) is then applied per pair.
2. 100% Rotational Router (GRTRotationalRouter, 0 nn.Linear Layers): Tokens are routed to experts via scaled cosine alignment between L2-normalized planar coordinate pairs (u_p, v_p) and E = 4 learned phase-angle prototypes (cos φ_{e,p}, sin φ_{e,p}) with learned softplus(output_scale) temperature.
3. Zero-Sum Walsh–Hadamard Upcycling & Multi-Domain Specialization (PairAdam): Initialized from MatterWaveAI/grt-30m via zero-sum Walsh–Hadamard angular perturbations (Σ_e Δ^(e) = 0) and specialized on a 4-domain corpus (rotational_geometry, science_and_physics, code_and_algorithms, reasoning_and_language) on an Intel Arc 140T XPU (xpu:0) with PairAdam and Switch-style load balancing:

Metric	Initial Upcycle	After MoR Specialization (PairAdam, XPU)	After MoR Specialization (PairAdam, ROCm gfx1151)
Held-Out Validation Loss	5.8467	0.5115	0.4829
Held-Out Validation Perplexity	346.10	1.67	1.62
Domain Loss — code_and_algorithms	5.4092	0.4334	0.3669
Domain Loss — rotational_geometry	6.6421	0.4991	0.4772 (0.4747 w/ PolarNaturalAdam)
Domain Loss — science_and_physics	5.3337	0.5293	0.5155
Domain Loss — reasoning_and_language	6.1720	0.6442	0.6466 (w/ PolarNaturalAdam)
Mean Expert Routing Load (E = 4)	[0.25, 0.25, 0.25, 0.25]	[0.250, 0.248, 0.256, 0.245]	[0.250, 0.250, 0.254, 0.246] (entropy = 1.9999 / 2.0000 bits)

---

8-Way Classical Geometric Optimizer Ablation (Spark ROCm gfx1151)

We benchmarked 8 classical geometric, symplectic, and gyroscopic optimizers from transformers.models.grt.optimization_grt on MatterWaveAI/grt-30m-mor (80 Phase-2 specialization steps on blocks[8:12] + final_norm from an identical Phase-1 cached prefix state, batch_size = 36, lr = 3.5e-3, max_angular_step = 0.08, seed = 42):

Rank	Optimizer (optimizer_type)	Held-Out Val Loss	Held-Out Val PPL	code_and_algorithms Loss	rotational_geometry Loss	science_and_physics Loss	reasoning_and_language Loss	Routing Entropy (bits / 2.0000)	Top-4 Expert Angle L2 Div	All-Layer Angle L2 Div	Time (s)
1	PairAdam (pair_adam)	0.4829	1.6208	0.3669	0.4772	0.5155	0.6553	1.9999 (1.3862 nats)	2.5347	2.5386	180.15s
2	PolarNaturalAdam (polar_natural_adam)	0.4848	1.6239	0.3744	0.4747	0.5226	0.6466	1.9999 (1.3862 nats)	2.5349	2.5387	180.75s
3	CyclideAdam (cyclide_adam)	0.5006	1.6498	0.3758	0.4986	0.5426	0.6718	1.9999 (1.3862 nats)	2.5789	2.5534	180.09s
4	HuygensAdam (huygens_adam)	0.5225	1.6862	0.3957	0.5107	0.5696	0.7041	1.9999 (1.3862 nats)	2.6216	2.5676	176.07s
5	HuygensSymplecticAdam (huygens_symplectic_adam)	0.5230	1.6871	0.4091	0.5202	0.5570	0.6859	1.9999 (1.3862 nats)	2.5351	2.5388	176.01s
6	HessianPairAdam (hessian_pair_adam)	0.5403	1.7165	0.4082	0.5330	0.5916	0.7201	1.9999 (1.3862 nats)	2.5349	2.5387	180.09s
7	SymplecticAdam (symplectic_adam)	0.5567	1.7448	0.4185	0.5530	0.6082	0.7417	1.9999 (1.3862 nats)	2.5345	2.5386	180.20s
8	GyricAdam (gyric_adam)	0.5745	1.7763	0.4279	0.5672	0.6412	0.7609	1.9999 (1.3862 nats)	2.5350	2.5387	177.29s

Key Geometric Findings on Mixture-of-Rotations (ffn_n_experts = 4)

• Huygens Circular Order Parameter Splay Torque Prevents Expert Collapse: On 3D MoR expert tensors (expert_expand_angles [4, 21, 512] and expert_contract_angles [4, 21, 512]), HuygensAdam computes the circular synchronization order parameter R_p = |(1/4) ∑_{e=0}^3 exp(i θ_{e,k,p})| directly across dimension 0 (E = 4 experts) and applies a zero-mean splay restoring torque whenever experts over-synchronize, achieving the highest pairwise expert angle L2 divergence (2.6216 in top-4 MoR blocks, 2.5676 all-layer mean) while maintaining 1.9999 / 2.0000 bits of routing entropy.
• Polar Riemannian Metric Excels on Conceptual & Geometric Domains: PolarNaturalAdam achieves #1 validation loss on rotational_geometry (0.4747) and #1 validation loss on reasoning_and_language (0.6466), finishing within 0.0019 overall loss of PairAdam (0.4848 vs. 0.4829).
• Curriculum Advantage of HuygensSymplecticAdam: Combining early-stage Huygens expert splay repulsion with late-stage Gromov–Liouville symplectic non-squeezing (HuygensSymplecticAdam, 0.5230 loss / 1.6871 PPL) outperforms standalone SymplecticAdam (0.5567), HessianPairAdam (0.5403), and GyricAdam (0.5745).

---

Quickstart (transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-mor"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)

inputs = tokenizer("The Givens Rotational Transformer replaces dense matrices with", return_tensors="pt")
out_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out_ids[0], skip_special_tokens=True))

Quickstart (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(model="MatterWaveAI/grt-30m-mor", model_impl="transformers", dtype="float32")
outputs = llm.generate(
    ["The Givens Rotational Transformer replaces dense matrices with"],
    SamplingParams(temperature=0.0, max_tokens=32),
)
print(outputs[0].outputs[0].text)

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer
8. Rotational Text-to-Speech (GRTTTS): MatterWaveAI/grt-30m-tts
9. Multimodal Vision-Audio-Language-Speech (GRTOmni): MatterWaveAI/grt-30m-omni

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}