| ---
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - grt
|
| - lora
|
| - peft
|
| - toroidal-lora
|
| - rotational-transformer
|
| - givens-rotations
|
| - clifford-algebra
|
| - sophia-torus
|
| - matterwave
|
| base_model: MatterWaveAI/grt-30m
|
| ---
|
|
|
| # GRT-30M-LoRA (`MatterWaveAI/grt-30m-lora`)
|
|
|
| **GRT-30M-LoRA** is the canonical **Toroidal Low-Rank Adaptation (`GRTRotationalLoRA`)** release for [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` base parameters, **`0` `nn.Linear` layers**, **`0` `nn.Conv1d` layers**).
|
|
|
| Because every projection in a Geometric Rotation Transformer is parameterized by phase angles on the flat torus `T^P = (S¹)^P` and radial scales rather than dense Euclidean weight matrices, standard linear LoRA (`W + B A`) cannot be applied without violating orthogonal/unitary manifold invariants. **`GRTRotationalLoRA`** instead attaches low-rank phase-shift matrices `lora_A` and `lora_B` directly to the butterfly rotation angle tensors:
|
|
|
| ```text
|
| θ = (θ_base + (α / r) · A B) mod 2π
|
| ```
|
|
|
| At inference time, `merge_grt_lora(model)` folds `Δθ = (α / r) A B` directly into `θ_base` with **bit-exact equivalence** (`max_abs_diff = 0.0`) and **zero runtime latency or parameter overhead**.
|
|
|
| ---
|
|
|
| ## Key Highlights & Metrics
|
|
|
| | Metric | Value |
|
| | :--- | :--- |
|
| | **Adapter Class** | `GRTRotationalLoRA` (`transformers.models.grt.lora_grt`) |
|
| | **Base Checkpoint** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` params) |
|
| | **LoRA Rank (`r`) / Alpha (`α`)** | `r = 8`, `lora_alpha = 16.0` (`scaling = 2.0`) |
|
| | **Trainable LoRA + Scale Parameters** | **`140,741`** (`0.48%` of base model) |
|
| | **Adapter Checkpoint Size** | **`573.3 KB`** (`grt_lora_adapter.safetensors`) |
|
| | **`nn.Linear` / `nn.Conv1d` Layers** | **`0` / `0` (100% Rotational Purity)** |
|
| | **Optimizer** | `HessianPairAdam` (`Sophia-Torus`) / **`SymplecticAdam` (`#1` in 8-way ablation)** |
|
| | **Domain Assistant Loss (Before → After)** | `6.1720` → **`4.7464`** (`HessianPairAdam`) / **`3.9882`** (`SymplecticAdam`) |
|
| | **Domain Perplexity (Before → After)** | `479.14` → **`115.16`** (`HessianPairAdam`) / **`53.96`** (`SymplecticAdam`) |
|
| | **Token Accuracy (Before → After)** | `19.5%` → **`28.8%`** (`HessianPairAdam`) / **`35.3%`** (`SymplecticAdam`) |
|
| | **Pre-Merge vs. Post-Merge `max_abs_diff`** | **`0.0e+00`** (bit-exact across all 8 optimizers) |
|
|
|
| ---
|
|
|
| ## 8-Way Classical Geometric Optimizer Ablation (`Spark ROCm gfx1151`)
|
|
|
| Because `GRTRotationalLoRA` parametrizes angle tensors (`q_angles`, `k_angles`, `v_angles`, `o_angles`, `coupling_angles`, `expand_angles`, `contract_angles`) with low-rank factors `lora_A` (`[K, r]`) and `lora_B` (`[r, P]`), `build_grt_optimizer` automatically routes all `131,232` LoRA matrix parameters into the toroidal angle group (`is_angle = True`, `weight_decay = 0.0`) and pairs them with their base module's radial scale semi-axes.
|
|
|
| We ran a controlled 8-way optimizer ablation on `MatterWaveAI/grt-30m-lora` (`r = 8`, `lora_alpha = 16.0`, `tune_scales = True`, `140,741` trainable parameters, `80` steps: `10` full-depth + `70` cached-prefix top-2-block steps, `lr = 1.8e-2`, `max_angular_step = 0.25`, `CosineAnnealingLR`, `seed = 2026`):
|
|
|
| | Rank | Optimizer (`optimizer_type`) | Before Loss | After Assistant Loss | Before PPL | After Perplexity | Before Token Acc | After Token Acc | Merge `max_abs_diff` | Time (s) |
|
| | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| | **1** | **`SymplecticAdam` (`symplectic_adam`)** | `6.1720` | **`3.9882`** | `479.14` | **`53.96`** | `19.54%` | **`35.34%`** | `0.0e+00` | `13.74s` |
|
| | **2** | **`GyricAdam` (`gyric_adam`)** | `6.1720` | `4.3119` | `479.14` | `74.58` | `19.54%` | `31.76%` | `0.0e+00` | `13.82s` |
|
| | **3** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `6.1720` | `4.5078` | `479.14` | `90.72` | `19.54%` | `30.78%` | `0.0e+00` | `13.34s` |
|
| | **4** | **`HessianPairAdam` (`hessian_pair_adam`)** | `6.1720` | `4.5159` | `479.14` | `91.46` | `19.54%` | `30.62%` | `0.0e+00` | `13.26s` |
|
| | **5** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | `6.1720` | `4.5764` | `479.14` | `97.17` | `19.54%` | `31.27%` | `0.0e+00` | `14.22s` |
|
| | **6** | **`CyclideAdam` (`cyclide_adam`)** | `6.1720` | `4.6722` | `479.14` | `106.93` | `19.54%` | `28.99%` | `0.0e+00` | `13.72s` |
|
| | **7** | **`HuygensAdam` (`huygens_adam`)** | `6.1720` | `4.9113` | `479.14` | `135.81` | `19.54%` | `25.41%` | `0.0e+00` | `13.50s` |
|
| | **8** | **`PairAdam` (`pair_adam`)** | `6.1720` | `5.0931` | `479.14` | `162.89` | `19.54%` | `21.34%` | `0.0e+00` | **`12.96s`** |
|
|
|
| ### Key Geometric Findings on `GRTRotationalLoRA`
|
| - **`SymplecticAdam` Achieves SOTA on Toroidal LoRA (`53.96` PPL / `35.34%` Token Accuracy)**: Because `lora_B` (`[r, P] = [8, P]`) is initialized to exact zeros (`0.0`) so that `Δθ = 0` at step 0, early gradients on `lora_A` and `lora_B` experience strong singular-value anisotropy across the `r = 8` low-rank bottleneck. `SymplecticAdam`'s Gromov–Liouville reciprocal non-squeezing stiffness floor (`zpe_floor`), Helmholtz–Casimir wave-cavity Laplacian smoothing across stages/ranks, and smooth Cayley rational retraction reduce domain perplexity by **`66.9%` relative to `PairAdam`** (`53.96` vs. `162.89`) and by **`41.0%` relative to `HessianPairAdam`** (`53.96` vs. `91.46`), while boosting token accuracy from `21.34%` (`PairAdam`) and `30.62%` (`HessianPairAdam`) to **`35.34%`**.
|
| - **`GyricAdam` Ranks Strong #2 (`74.58` PPL / `31.76%` Token Accuracy)**: Poinsot polar-conic inertia preconditioning combined with Landé vector-model projection and retrograde Thomas precession damps high-curvature rank oscillations in `lora_A @ lora_B`.
|
| - **`HuygensSymplecticAdam` Ranks #3 in Token Accuracy (`31.27%`)**: Annealing the early Huygens rank-splay perturbation into late-stage symplectic non-squeezing recovers `+5.86%` token accuracy over standalone `HuygensAdam` (`31.27%` vs. `25.41%`).
|
|
|
| ---
|
|
|
| ## Quickstart (`transformers`)
|
|
|
| ### 1. Load & Merge `MatterWaveAI/grt-30m-lora` into `MatterWaveAI/grt-30m`
|
|
|
| ```python
|
| import torch
|
| from huggingface_hub import snapshot_download
|
| from transformers import AutoTokenizer
|
| from transformers.models.grt import GRTForCausalLM, load_grt_lora_pretrained
|
|
|
| base_id = "MatterWaveAI/grt-30m"
|
| adapter_dir = snapshot_download("MatterWaveAI/grt-30m-lora")
|
|
|
| tokenizer = AutoTokenizer.from_pretrained(base_id)
|
| model = GRTForCausalLM.from_pretrained(base_id, torch_dtype=torch.float32)
|
|
|
| # Attach and fold the toroidal LoRA phase shifts into base_angles (0 inference overhead)
|
| load_grt_lora_pretrained(model, adapter_dir, merge=True)
|
|
|
| prompt = "<|im_start|>user\nExplain how GRTRotationalLoRA adapts a Geometric Rotation Transformer without linear layers.<|im_end|>\n<|im_start|>assistant\n"
|
| inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
|
| out_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)
|
| print(tokenizer.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| ```
|
|
|
| ### 2. Train Your Own Toroidal LoRA Adapter
|
|
|
| ```python
|
| from transformers.models.grt import (
|
| GRTForCausalLM,
|
| attach_grt_lora,
|
| build_grt_optimizer,
|
| merge_grt_lora,
|
| save_grt_lora_pretrained,
|
| )
|
|
|
| model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-30m")
|
| attach_grt_lora(model, r=8, lora_alpha=16.0, tune_scales=True)
|
|
|
| optimizer = build_grt_optimizer(model, lr=1e-2, weight_decay=0.0, hessian_mode=True)
|
| # ... training loop ...
|
| save_grt_lora_pretrained(model, "./my-grt-lora")
|
| merge_grt_lora(model) # zero-overhead merged inference
|
| ```
|
|
|
| ---
|
|
|
| ## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
|
|
|
| 1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
|
| 2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
|
| 3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
|
| 4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
|
| 5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
|
| 6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
|
| 7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
|
| 8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
|
| 9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)
|
| 10. **Toroidal Rotational LoRA (`GRTRotationalLoRA`)**: [`MatterWaveAI/grt-30m-lora`](https://huggingface.co/MatterWaveAI/grt-30m-lora)
|
| 11. **Rotational Speech-to-Text ASR (`GRTASR`)**: [`MatterWaveAI/grt-30m-asr`](https://huggingface.co/MatterWaveAI/grt-30m-asr)
|
|
|
| - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
|
|
|
| ---
|
|
|
| ## License
|
|
|
| Apache 2.0
|
|
|
| ---
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @techreport{householter2026grt,
|
| title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
|
| author = {Householter, David},
|
| institution = {MatterWave AI},
|
| year = {2026},
|
| url = {https://matterwave.ai}
|
| }
|
| ```
|
|
|
