---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - grt
  - lora
  - peft
  - toroidal-lora
  - rotational-transformer
  - givens-rotations
  - clifford-algebra
  - sophia-torus
  - matterwave
base_model: MatterWaveAI/grt-30m
---

# GRT-30M-LoRA (`MatterWaveAI/grt-30m-lora`)

**GRT-30M-LoRA** is the canonical **Toroidal Low-Rank Adaptation (`GRTRotationalLoRA`)** release for [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` base parameters, **`0` `nn.Linear` layers**, **`0` `nn.Conv1d` layers**).

Because every projection in a Geometric Rotation Transformer is parameterized by phase angles on the flat torus `T^P = (S¹)^P` and radial scales rather than dense Euclidean weight matrices, standard linear LoRA (`W + B A`) cannot be applied without violating orthogonal/unitary manifold invariants. **`GRTRotationalLoRA`** instead attaches low-rank phase-shift matrices `lora_A` and `lora_B` directly to the butterfly rotation angle tensors:

```text
θ = (θ_base + (α / r) · A B) mod 2π
```

At inference time, `merge_grt_lora(model)` folds `Δθ = (α / r) A B` directly into `θ_base` with **bit-exact equivalence** (`max_abs_diff = 0.0`) and **zero runtime latency or parameter overhead**.

---

## Key Highlights & Metrics

| Metric | Value |
| :--- | :--- |
| **Adapter Class** | `GRTRotationalLoRA` (`transformers.models.grt.lora_grt`) |
| **Base Checkpoint** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` params) |
| **LoRA Rank (`r`) / Alpha (`α`)** | `r = 8`, `lora_alpha = 16.0` (`scaling = 2.0`) |
| **Trainable LoRA + Scale Parameters** | **`140,741`** (`0.48%` of base model) |
| **Adapter Checkpoint Size** | **`573.3 KB`** (`grt_lora_adapter.safetensors`) |
| **`nn.Linear` / `nn.Conv1d` Layers** | **`0` / `0` (100% Rotational Purity)** |
| **Optimizer** | `HessianPairAdam` (`Sophia-Torus`) / **`SymplecticAdam` (`#1` in 8-way ablation)** |
| **Domain Assistant Loss (Before → After)** | `6.1720` → **`4.7464`** (`HessianPairAdam`) / **`3.9882`** (`SymplecticAdam`) |
| **Domain Perplexity (Before → After)** | `479.14` → **`115.16`** (`HessianPairAdam`) / **`53.96`** (`SymplecticAdam`) |
| **Token Accuracy (Before → After)** | `19.5%` → **`28.8%`** (`HessianPairAdam`) / **`35.3%`** (`SymplecticAdam`) |
| **Pre-Merge vs. Post-Merge `max_abs_diff`** | **`0.0e+00`** (bit-exact across all 8 optimizers) |

---

## 8-Way Classical Geometric Optimizer Ablation (`Spark ROCm gfx1151`)

Because `GRTRotationalLoRA` parametrizes angle tensors (`q_angles`, `k_angles`, `v_angles`, `o_angles`, `coupling_angles`, `expand_angles`, `contract_angles`) with low-rank factors `lora_A` (`[K, r]`) and `lora_B` (`[r, P]`), `build_grt_optimizer` automatically routes all `131,232` LoRA matrix parameters into the toroidal angle group (`is_angle = True`, `weight_decay = 0.0`) and pairs them with their base module's radial scale semi-axes.

We ran a controlled 8-way optimizer ablation on `MatterWaveAI/grt-30m-lora` (`r = 8`, `lora_alpha = 16.0`, `tune_scales = True`, `140,741` trainable parameters, `80` steps: `10` full-depth + `70` cached-prefix top-2-block steps, `lr = 1.8e-2`, `max_angular_step = 0.25`, `CosineAnnealingLR`, `seed = 2026`):

| Rank | Optimizer (`optimizer_type`) | Before Loss | After Assistant Loss | Before PPL | After Perplexity | Before Token Acc | After Token Acc | Merge `max_abs_diff` | Time (s) |
| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **1** | **`SymplecticAdam` (`symplectic_adam`)** | `6.1720` | **`3.9882`** | `479.14` | **`53.96`** | `19.54%` | **`35.34%`** | `0.0e+00` | `13.74s` |
| **2** | **`GyricAdam` (`gyric_adam`)** | `6.1720` | `4.3119` | `479.14` | `74.58` | `19.54%` | `31.76%` | `0.0e+00` | `13.82s` |
| **3** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `6.1720` | `4.5078` | `479.14` | `90.72` | `19.54%` | `30.78%` | `0.0e+00` | `13.34s` |
| **4** | **`HessianPairAdam` (`hessian_pair_adam`)** | `6.1720` | `4.5159` | `479.14` | `91.46` | `19.54%` | `30.62%` | `0.0e+00` | `13.26s` |
| **5** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | `6.1720` | `4.5764` | `479.14` | `97.17` | `19.54%` | `31.27%` | `0.0e+00` | `14.22s` |
| **6** | **`CyclideAdam` (`cyclide_adam`)** | `6.1720` | `4.6722` | `479.14` | `106.93` | `19.54%` | `28.99%` | `0.0e+00` | `13.72s` |
| **7** | **`HuygensAdam` (`huygens_adam`)** | `6.1720` | `4.9113` | `479.14` | `135.81` | `19.54%` | `25.41%` | `0.0e+00` | `13.50s` |
| **8** | **`PairAdam` (`pair_adam`)** | `6.1720` | `5.0931` | `479.14` | `162.89` | `19.54%` | `21.34%` | `0.0e+00` | **`12.96s`** |

### Key Geometric Findings on `GRTRotationalLoRA`
- **`SymplecticAdam` Achieves SOTA on Toroidal LoRA (`53.96` PPL / `35.34%` Token Accuracy)**: Because `lora_B` (`[r, P] = [8, P]`) is initialized to exact zeros (`0.0`) so that `Δθ = 0` at step 0, early gradients on `lora_A` and `lora_B` experience strong singular-value anisotropy across the `r = 8` low-rank bottleneck. `SymplecticAdam`'s Gromov–Liouville reciprocal non-squeezing stiffness floor (`zpe_floor`), Helmholtz–Casimir wave-cavity Laplacian smoothing across stages/ranks, and smooth Cayley rational retraction reduce domain perplexity by **`66.9%` relative to `PairAdam`** (`53.96` vs. `162.89`) and by **`41.0%` relative to `HessianPairAdam`** (`53.96` vs. `91.46`), while boosting token accuracy from `21.34%` (`PairAdam`) and `30.62%` (`HessianPairAdam`) to **`35.34%`**.
- **`GyricAdam` Ranks Strong #2 (`74.58` PPL / `31.76%` Token Accuracy)**: Poinsot polar-conic inertia preconditioning combined with Landé vector-model projection and retrograde Thomas precession damps high-curvature rank oscillations in `lora_A @ lora_B`.
- **`HuygensSymplecticAdam` Ranks #3 in Token Accuracy (`31.27%`)**: Annealing the early Huygens rank-splay perturbation into late-stage symplectic non-squeezing recovers `+5.86%` token accuracy over standalone `HuygensAdam` (`31.27%` vs. `25.41%`).

---

## Quickstart (`transformers`)

### 1. Load & Merge `MatterWaveAI/grt-30m-lora` into `MatterWaveAI/grt-30m`

```python
import torch
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
from transformers.models.grt import GRTForCausalLM, load_grt_lora_pretrained

base_id = "MatterWaveAI/grt-30m"
adapter_dir = snapshot_download("MatterWaveAI/grt-30m-lora")

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = GRTForCausalLM.from_pretrained(base_id, torch_dtype=torch.float32)

# Attach and fold the toroidal LoRA phase shifts into base_angles (0 inference overhead)
load_grt_lora_pretrained(model, adapter_dir, merge=True)

prompt = "<|im_start|>user\nExplain how GRTRotationalLoRA adapts a Geometric Rotation Transformer without linear layers.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
out_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

### 2. Train Your Own Toroidal LoRA Adapter

```python
from transformers.models.grt import (
    GRTForCausalLM,
    attach_grt_lora,
    build_grt_optimizer,
    merge_grt_lora,
    save_grt_lora_pretrained,
)

model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-30m")
attach_grt_lora(model, r=8, lora_alpha=16.0, tune_scales=True)

optimizer = build_grt_optimizer(model, lr=1e-2, weight_decay=0.0, hessian_mode=True)
# ... training loop ...
save_grt_lora_pretrained(model, "./my-grt-lora")
merge_grt_lora(model)  # zero-overhead merged inference
```

---

## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face

1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)
10. **Toroidal Rotational LoRA (`GRTRotationalLoRA`)**: [`MatterWaveAI/grt-30m-lora`](https://huggingface.co/MatterWaveAI/grt-30m-lora)
11. **Rotational Speech-to-Text ASR (`GRTASR`)**: [`MatterWaveAI/grt-30m-asr`](https://huggingface.co/MatterWaveAI/grt-30m-asr)

- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)

---

## License

Apache 2.0

---

## Citation

```bibtex
@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}
```