---
license: apache-2.0
tags:
  - grt
  - geometric-rotation-transformer
  - causal-lm
  - rotational
datasets:
  - nvidia/Nemotron-Pretraining-Dataset-sample
language:
  - en
library_name: transformers
pipeline_tag: text-generation
---

# GRT-10M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization

> **Note:** `GRT-10M` (`8.7M` total parameters, `2.2M` non-embedding backbone parameters) was originally built as the small-scale proof-of-concept for our NeurIPS submission using first-generation `PairAdam` (archived pre-fine-tune at [`MatterWaveAI/grt-10m-base`](https://huggingface.co/MatterWaveAI/grt-10m-base)). Keeping the exact same `8.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`37.61%` (`PairAdam`) → `41.71%` (`HuygensSymplecticAdam`)** and cuts `GSM8K` math perplexity from **`145.9` → `30.1` (`4.85×` reduction)**. For the multi-round curriculum checkpoint (**`45.74%`** 9-task MC average, **`96.60%`** `GSM8K` verification, **`25.28`** PPL), see [`MatterWaveAI/grt-10m-curriculum`](https://huggingface.co/MatterWaveAI/grt-10m-curriculum).

A Geometric Rotation Transformer pretrained on Nemotron. GRT replaces the dense
linear projections of a standard transformer with cascades of learned Givens
rotations on SO(D), so the residual path contains no `nn.Linear` at all.

This is a research proof-of-concept at small scale. It is published to
demonstrate that the architecture trains stably and evaluates end-to-end, not
as a capable language model.

## Model Details

| | |
|---|---|
| **Total parameters** | 8,685,283 (8.7M) |
| **Non-embedding parameters** | 2,235,283 (2.2M) |
| **d_model** | 256 |
| **n_blocks** | 6 |
| **n_heads** | 4 |
| **n_qkv_rotations** | 5 |
| **ffn_n_rotations** | 9 |
| **Vocabulary** | 50,000 (BPE) |
| **Max sequence length** | 1,024 |
| **Precision** | float32 |

The embedding dominates the parameter count: 6.45M of 8.7M parameters are the
token embedding (50,000 tokens x 128 phase angles, plus a log-magnitude per
token).

## Training

| | |
|---|---|
| **Base Pretraining Data** | Nemotron |
| **Base Pretraining Steps** | 336,000 |
| **Base Pretraining Optimizer** | PairAdam (`MatterWaveAI/grt-10m-base`) |
| **Base Peak LR** | 1e-3, cosine decay to 10% |
| **Base Training Loss** | 2.144 |
| **Task Adaptation Optimizer** | `HuygensSymplecticAdam` (`SymplecticAdam` two-stage curriculum) |

| Metric | Value |
|---|---|
| Validation loss | 4.033 |
| Validation perplexity | 56.4 |
| Validation accuracy | 0.400 |

## Benchmarks: NeurIPS Submission Baseline (`PairAdam`) vs. New Optimizer (`HuygensSymplecticAdam`)

Evaluated on the `SLM-Bench` small-language-model suite (`training/slm_bench.py` / `grt-slm-bench`) starting from the exact same `8.7M`-parameter (`2.2M` non-embedding) NeurIPS-submission checkpoint:

### 1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)

| Task (`SLM-Bench`) | Split (`n`) | Random | `GRT-10M` Zero-Shot | NeurIPS Baseline (`PairAdam`) | `SymplecticAdam` | New Optimizer (`HuygensSymplecticAdam`) | Gain vs. `PairAdam` |
|---|---|---:|---:|---:|---:|---:|---:|
| **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `62.14%` | `61.13%` | `62.14%` | **`62.17%`** | **`+1.04%`** (`+12.17%` vs. random) |
| **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `52.70%` | `49.70%` | `56.80%` | **`60.90%`** | **`+11.20%`** |
| **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | `52.18%` | `50.82%` | **`52.18%`** | **`52.18%`** | **`+1.36%`** |
| **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.51%` | `50.75%` | **`51.85%`** | **`51.85%`** | **`+1.10%`** |
| **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `30.00%` | `22.60%` | `33.40%` | **`35.20%`** | **`+12.60%`** |
| **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `26.94%` | `26.60%` | `31.73%` | **`32.28%`** | **`+5.68%`** |
| **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `25.60%` | `29.10%` | **`30.80%`** | **`30.80%`** | **`+1.70%`** |
| **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `22.53%` | `24.91%` | `25.17%` | **`27.13%`** | **`+2.22%`** |
| **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `20.15%` | `22.85%` | `22.85%` | **`22.85%`** | **`+2.70%`** vs. zero-shot |
| **9-Task Multiple-Choice Macro Average** | `13,144` total | `35.56%` | `38.08%` | `37.61%` | `40.77%` | **`41.71%`** | **`+4.10%`** (`+6.15%` vs. random) |

### 2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`)

| Task (`SLM-Bench`) | Split (`n`) | `GRT-10M` Zero-Shot | NeurIPS Baseline (`PairAdam`) | `SymplecticAdam` | New Optimizer (`HuygensSymplecticAdam`) | Improvement |
|---|---|---:|---:|---:|---:|---:|
| **GSM8K Held-Out Perplexity (`PPL` ↓)** | `test` (`500`) | `145.85` | `37.27` | `43.47` | **`30.09`** | **`4.85×` lower vs. Zero-Shot (`-19.3%` vs. `PairAdam`)** |
| **GSM8K 4-Way Numerical Verification (`Acc` ↑)** | `test` (`500`) | `22.00%` | `21.80%` | `79.40%` | **`84.60%`** | **`+62.80%` vs. `PairAdam`** |

Reproduce with `grt-slm-bench`:

```bash
python -m training.slm_bench --checkpoint ./ --optimizer huygens_symplectic_adam \
    --tasks boolq,arc_easy,arc_challenge,openbookqa,piqa,winogrande,hellaswag,qnli,commonsenseqa,gsm8k \
    --device cuda
```

Zero-shot, via lm-evaluation-harness:

```bash
lm_eval --model hf --model_args pretrained=MatterWaveAI/grt-10m \
    --tasks boolq,arc_easy --batch_size 64
```

## Usage

```bash
pip install geometric-rotation-transformer
```

```python
from grt import GRTForCausalLM, RotationalTokenizerBPE

model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-10m")
tokenizer = RotationalTokenizerBPE.from_pretrained("MatterWaveAI/grt-10m")

input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Importing `grt` registers the architecture with the `transformers` Auto classes:

```python
import grt  # registers model_type "grt"
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-10m")
```

The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional
and load automatically when installed.

> **`trust_remote_code=True` is not supported for this repository.** The model
> code is distributed as a pip package rather than as Hub remote code, because
> the implementation spans a package with absolute imports that a flat Hub
> snapshot cannot resolve. Install `geometric-rotation-transformer` first;
> `import grt` then registers `model_type: "grt"` so the Auto classes resolve
> these weights.

## Architecture

GRT replaces the standard Q/K/V projections and FFN weight matrices with
cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is
modulated by data-dependent Clifford features:

```
feat  = [S0, V1, V2, B12]           # scalar energy, direction, cross-pair bivector
theta = angles + tanh(modulation)   # base angle + content-dependent nudge
r0    = x0*cos(theta) - x1*sin(theta)
r1    = x0*sin(theta) + x1*cos(theta)
```

Components:

- **RotationalEmbedding** — tokens stored as phase angles on the unit hypersphere
- **RotationalAttention** — Q/K/V via Givens cascades
- **RotationalFFN** — feed-forward via meta-coupled rotation stages
- **PearlNorm** — pair-radius normalization preserving per-pair phase
- **RadialGELU** — activation gating pair magnitudes, preserving direction
- **PairAdam** — AdamW with factored second moments: 1D params (angles, scales)
  use standard diagonal moments, 2D params (coupling matrices) use a row factor
  [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n)
  instead of O(mn). Optionally clamps per-step angular updates.

Because every projection is a rotation, the transform is norm-preserving by
construction rather than by normalization.

## Intended Use & Limitations

Intended for research into structured/rotational architectures. Not intended for
production use or deployment.

Known limitations:

- English only, trained on a single corpus (Nemotron), so it inherits that
  corpus's coverage gaps and biases.
- No instruction tuning. This is a base model trained on next-token prediction.
- At 8.7M parameters, generated text is frequently incoherent. Do not expect
  instruction-following, factual reliability, or coherent long-form output.
- No safety tuning, RLHF, or content filtering of any kind.
- Context is capped at 1,024 tokens.

## Citation

```bibtex
@article{grt2026,
  title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling},
  author={MatterWave AI},
  year={2026},
  url={https://github.com/matterwaveai/grt}
}
```

## License

Apache 2.0