GRT-30M-LoRA (MatterWaveAI/grt-30m-lora)

GRT-30M-LoRA is the canonical Toroidal Low-Rank Adaptation (GRTRotationalLoRA) release for MatterWaveAI/grt-30m (29,305,498 base parameters, 0 nn.Linear layers, 0 nn.Conv1d layers).

Because every projection in a Geometric Rotation Transformer is parameterized by phase angles on the flat torus T^P = (S¹)^P and radial scales rather than dense Euclidean weight matrices, standard linear LoRA (W + B A) cannot be applied without violating orthogonal/unitary manifold invariants. GRTRotationalLoRA instead attaches low-rank phase-shift matrices lora_A and lora_B directly to the butterfly rotation angle tensors:

θ = (θ_base + (α / r) · A B) mod 2π

At inference time, merge_grt_lora(model) folds Δθ = (α / r) A B directly into θ_base with bit-exact equivalence (max_abs_diff = 0.0) and zero runtime latency or parameter overhead.

---

Key Highlights & Metrics

Metric	Value
Adapter Class	GRTRotationalLoRA (transformers.models.grt.lora_grt)
Base Checkpoint	MatterWaveAI/grt-30m (29,305,498 params)
LoRA Rank (r) / Alpha (α)	r = 8, lora_alpha = 16.0 (scaling = 2.0)
Trainable LoRA + Scale Parameters	140,741 (0.48% of base model)
Adapter Checkpoint Size	573.3 KB (grt_lora_adapter.safetensors)
nn.Linear / nn.Conv1d Layers	0 / 0 (100% Rotational Purity)
Optimizer	HessianPairAdam (Sophia-Torus) / SymplecticAdam (#1 in 8-way ablation)
Domain Assistant Loss (Before → After)	6.1720 → 4.7464 (HessianPairAdam) / 3.9882 (SymplecticAdam)
Domain Perplexity (Before → After)	479.14 → 115.16 (HessianPairAdam) / 53.96 (SymplecticAdam)
Token Accuracy (Before → After)	19.5% → 28.8% (HessianPairAdam) / 35.3% (SymplecticAdam)
Pre-Merge vs. Post-Merge max_abs_diff	0.0e+00 (bit-exact across all 8 optimizers)

---

8-Way Classical Geometric Optimizer Ablation (Spark ROCm gfx1151)

Because GRTRotationalLoRA parametrizes angle tensors (q_angles, k_angles, v_angles, o_angles, coupling_angles, expand_angles, contract_angles) with low-rank factors lora_A ([K, r]) and lora_B ([r, P]), build_grt_optimizer automatically routes all 131,232 LoRA matrix parameters into the toroidal angle group (is_angle = True, weight_decay = 0.0) and pairs them with their base module's radial scale semi-axes.

We ran a controlled 8-way optimizer ablation on MatterWaveAI/grt-30m-lora (r = 8, lora_alpha = 16.0, tune_scales = True, 140,741 trainable parameters, 80 steps: 10 full-depth + 70 cached-prefix top-2-block steps, lr = 1.8e-2, max_angular_step = 0.25, CosineAnnealingLR, seed = 2026):

Rank	Optimizer (optimizer_type)	Before Loss	After Assistant Loss	Before PPL	After Perplexity	Before Token Acc	After Token Acc	Merge max_abs_diff	Time (s)
1	SymplecticAdam (symplectic_adam)	6.1720	3.9882	479.14	53.96	19.54%	35.34%	0.0e+00	13.74s
2	GyricAdam (gyric_adam)	6.1720	4.3119	479.14	74.58	19.54%	31.76%	0.0e+00	13.82s
3	PolarNaturalAdam (polar_natural_adam)	6.1720	4.5078	479.14	90.72	19.54%	30.78%	0.0e+00	13.34s
4	HessianPairAdam (hessian_pair_adam)	6.1720	4.5159	479.14	91.46	19.54%	30.62%	0.0e+00	13.26s
5	HuygensSymplecticAdam (huygens_symplectic_adam)	6.1720	4.5764	479.14	97.17	19.54%	31.27%	0.0e+00	14.22s
6	CyclideAdam (cyclide_adam)	6.1720	4.6722	479.14	106.93	19.54%	28.99%	0.0e+00	13.72s
7	HuygensAdam (huygens_adam)	6.1720	4.9113	479.14	135.81	19.54%	25.41%	0.0e+00	13.50s
8	PairAdam (pair_adam)	6.1720	5.0931	479.14	162.89	19.54%	21.34%	0.0e+00	12.96s

Key Geometric Findings on GRTRotationalLoRA

• SymplecticAdam Achieves SOTA on Toroidal LoRA (53.96 PPL / 35.34% Token Accuracy): Because lora_B ([r, P] = [8, P]) is initialized to exact zeros (0.0) so that Δθ = 0 at step 0, early gradients on lora_A and lora_B experience strong singular-value anisotropy across the r = 8 low-rank bottleneck. SymplecticAdam's Gromov–Liouville reciprocal non-squeezing stiffness floor (zpe_floor), Helmholtz–Casimir wave-cavity Laplacian smoothing across stages/ranks, and smooth Cayley rational retraction reduce domain perplexity by 66.9% relative to PairAdam (53.96 vs. 162.89) and by 41.0% relative to HessianPairAdam (53.96 vs. 91.46), while boosting token accuracy from 21.34% (PairAdam) and 30.62% (HessianPairAdam) to 35.34%.
• GyricAdam Ranks Strong #2 (74.58 PPL / 31.76% Token Accuracy): Poinsot polar-conic inertia preconditioning combined with Landé vector-model projection and retrograde Thomas precession damps high-curvature rank oscillations in lora_A @ lora_B.
• HuygensSymplecticAdam Ranks #3 in Token Accuracy (31.27%): Annealing the early Huygens rank-splay perturbation into late-stage symplectic non-squeezing recovers +5.86% token accuracy over standalone HuygensAdam (31.27% vs. 25.41%).

---

Quickstart (transformers)

1. Load & Merge MatterWaveAI/grt-30m-lora into MatterWaveAI/grt-30m

import torch
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
from transformers.models.grt import GRTForCausalLM, load_grt_lora_pretrained

base_id = "MatterWaveAI/grt-30m"
adapter_dir = snapshot_download("MatterWaveAI/grt-30m-lora")

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = GRTForCausalLM.from_pretrained(base_id, torch_dtype=torch.float32)

# Attach and fold the toroidal LoRA phase shifts into base_angles (0 inference overhead)
load_grt_lora_pretrained(model, adapter_dir, merge=True)

prompt = "<|im_start|>user\nExplain how GRTRotationalLoRA adapts a Geometric Rotation Transformer without linear layers.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
out_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

2. Train Your Own Toroidal LoRA Adapter

from transformers.models.grt import (
    GRTForCausalLM,
    attach_grt_lora,
    build_grt_optimizer,
    merge_grt_lora,
    save_grt_lora_pretrained,
)

model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-30m")
attach_grt_lora(model, r=8, lora_alpha=16.0, tune_scales=True)

optimizer = build_grt_optimizer(model, lr=1e-2, weight_decay=0.0, hessian_mode=True)
# ... training loop ...
save_grt_lora_pretrained(model, "./my-grt-lora")
merge_grt_lora(model)  # zero-overhead merged inference

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer
8. Rotational Text-to-Speech (GRTTTS): MatterWaveAI/grt-30m-tts
9. Multimodal Vision-Audio-Language-Speech (GRTOmni): MatterWaveAI/grt-30m-omni
10. Toroidal Rotational LoRA (GRTRotationalLoRA): MatterWaveAI/grt-30m-lora
11. Rotational Speech-to-Text ASR (GRTASR): MatterWaveAI/grt-30m-asr

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}