GRT-10M: From NeurIPS Submission (PairAdam) to Symplectic/Huygens Optimization
Note:
GRT-10M(8.7Mtotal parameters,2.2Mnon-embedding backbone parameters) was originally built as the small-scale proof-of-concept for our NeurIPS submission using first-generationPairAdam(archived pre-fine-tune atMatterWaveAI/grt-10m-base). Keeping the exact same8.7MNeurIPS submission checkpoint and swapping in our new classical geometric optimizers (SymplecticAdamandHuygensSymplecticAdam) lifts 9-taskSLM-Benchmultiple-choice accuracy from37.61%(PairAdam) →41.71%(HuygensSymplecticAdam) and cutsGSM8Kmath perplexity from145.9→30.1(4.85×reduction). For the multi-round curriculum checkpoint (45.74%9-task MC average,96.60%GSM8Kverification,25.28PPL), seeMatterWaveAI/grt-10m-curriculum.
A Geometric Rotation Transformer pretrained on Nemotron. GRT replaces the dense
linear projections of a standard transformer with cascades of learned Givens
rotations on SO(D), so the residual path contains no nn.Linear at all.
This is a research proof-of-concept at small scale. It is published to demonstrate that the architecture trains stably and evaluates end-to-end, not as a capable language model.
Model Details
| Total parameters | 8,685,283 (8.7M) |
| Non-embedding parameters | 2,235,283 (2.2M) |
| d_model | 256 |
| n_blocks | 6 |
| n_heads | 4 |
| n_qkv_rotations | 5 |
| ffn_n_rotations | 9 |
| Vocabulary | 50,000 (BPE) |
| Max sequence length | 1,024 |
| Precision | float32 |
The embedding dominates the parameter count: 6.45M of 8.7M parameters are the token embedding (50,000 tokens x 128 phase angles, plus a log-magnitude per token).
Training
| Base Pretraining Data | Nemotron |
| Base Pretraining Steps | 336,000 |
| Base Pretraining Optimizer | PairAdam (MatterWaveAI/grt-10m-base) |
| Base Peak LR | 1e-3, cosine decay to 10% |
| Base Training Loss | 2.144 |
| Task Adaptation Optimizer | HuygensSymplecticAdam (SymplecticAdam two-stage curriculum) |
| Metric | Value |
|---|---|
| Validation loss | 4.033 |
| Validation perplexity | 56.4 |
| Validation accuracy | 0.400 |
Benchmarks: NeurIPS Submission Baseline (PairAdam) vs. New Optimizer (HuygensSymplecticAdam)
Evaluated on the SLM-Bench small-language-model suite (training/slm_bench.py / grt-slm-bench) starting from the exact same 8.7M-parameter (2.2M non-embedding) NeurIPS-submission checkpoint:
1. Multiple-Choice Accuracy (9 Core SLM-Bench Tasks, 13,144 Held-Out Examples)
Task (SLM-Bench) |
Split (n) |
Random | GRT-10M Zero-Shot |
NeurIPS Baseline (PairAdam) |
SymplecticAdam |
New Optimizer (HuygensSymplecticAdam) |
Gain vs. PairAdam |
|---|---|---|---|---|---|---|---|
BoolQ (google/boolq) |
val (3,270) |
50.00% |
62.14% |
61.13% |
62.14% |
62.17% |
+1.04% (+12.17% vs. random) |
QNLI (nyu-mll/glue) |
val (1,000) |
50.00% |
52.70% |
49.70% |
56.80% |
60.90% |
+11.20% |
PIQA (baber/piqa) |
val (1,838) |
50.00% |
52.18% |
50.82% |
52.18% |
52.18% |
+1.36% |
WinoGrande (allenai/winogrande) |
val (1,267) |
50.00% |
50.51% |
50.75% |
51.85% |
51.85% |
+1.10% |
OpenBookQA (allenai/openbookqa) |
test (500) |
25.00% |
30.00% |
22.60% |
33.40% |
35.20% |
+12.60% |
ARC-Easy (allenai/ai2_arc) |
test (2,376) |
25.00% |
26.94% |
26.60% |
31.73% |
32.28% |
+5.68% |
HellaSwag (Rowan/hellaswag) |
val (1,000) |
25.00% |
25.60% |
29.10% |
30.80% |
30.80% |
+1.70% |
ARC-Challenge (allenai/ai2_arc) |
test (1,172) |
25.00% |
22.53% |
24.91% |
25.17% |
27.13% |
+2.22% |
CommonsenseQA (tau/commonsense_qa) |
val (1,221) |
20.00% |
20.15% |
22.85% |
22.85% |
22.85% |
+2.70% vs. zero-shot |
| 9-Task Multiple-Choice Macro Average | 13,144 total |
35.56% |
38.08% |
37.61% |
40.77% |
41.71% |
+4.10% (+6.15% vs. random) |
2. Mathematical Reasoning Held-Out Perplexity & Verification (GSM8K)
Task (SLM-Bench) |
Split (n) |
GRT-10M Zero-Shot |
NeurIPS Baseline (PairAdam) |
SymplecticAdam |
New Optimizer (HuygensSymplecticAdam) |
Improvement |
|---|---|---|---|---|---|---|
GSM8K Held-Out Perplexity (PPL ↓) |
test (500) |
145.85 |
37.27 |
43.47 |
30.09 |
4.85× lower vs. Zero-Shot (-19.3% vs. PairAdam) |
GSM8K 4-Way Numerical Verification (Acc ↑) |
test (500) |
22.00% |
21.80% |
79.40% |
84.60% |
+62.80% vs. PairAdam |
Reproduce with grt-slm-bench:
python -m training.slm_bench --checkpoint ./ --optimizer huygens_symplectic_adam \
--tasks boolq,arc_easy,arc_challenge,openbookqa,piqa,winogrande,hellaswag,qnli,commonsenseqa,gsm8k \
--device cuda
Zero-shot, via lm-evaluation-harness:
lm_eval --model hf --model_args pretrained=MatterWaveAI/grt-10m \
--tasks boolq,arc_easy --batch_size 64
Usage
pip install geometric-rotation-transformer
from grt import GRTForCausalLM, RotationalTokenizerBPE
model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-10m")
tokenizer = RotationalTokenizerBPE.from_pretrained("MatterWaveAI/grt-10m")
input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Importing grt registers the architecture with the transformers Auto classes:
import grt # registers model_type "grt"
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-10m")
The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional and load automatically when installed.
trust_remote_code=Trueis not supported for this repository. The model code is distributed as a pip package rather than as Hub remote code, because the implementation spans a package with absolute imports that a flat Hub snapshot cannot resolve. Installgeometric-rotation-transformerfirst;import grtthen registersmodel_type: "grt"so the Auto classes resolve these weights.
Architecture
GRT replaces the standard Q/K/V projections and FFN weight matrices with cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is modulated by data-dependent Clifford features:
feat = [S0, V1, V2, B12] # scalar energy, direction, cross-pair bivector
theta = angles + tanh(modulation) # base angle + content-dependent nudge
r0 = x0*cos(theta) - x1*sin(theta)
r1 = x0*sin(theta) + x1*cos(theta)
Components:
- RotationalEmbedding — tokens stored as phase angles on the unit hypersphere
- RotationalAttention — Q/K/V via Givens cascades
- RotationalFFN — feed-forward via meta-coupled rotation stages
- PearlNorm — pair-radius normalization preserving per-pair phase
- RadialGELU — activation gating pair magnitudes, preserving direction
- PairAdam — AdamW with factored second moments: 1D params (angles, scales) use standard diagonal moments, 2D params (coupling matrices) use a row factor [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n) instead of O(mn). Optionally clamps per-step angular updates.
Because every projection is a rotation, the transform is norm-preserving by construction rather than by normalization.
Intended Use & Limitations
Intended for research into structured/rotational architectures. Not intended for production use or deployment.
Known limitations:
- English only, trained on a single corpus (Nemotron), so it inherits that corpus's coverage gaps and biases.
- No instruction tuning. This is a base model trained on next-token prediction.
- At 8.7M parameters, generated text is frequently incoherent. Do not expect instruction-following, factual reliability, or coherent long-form output.
- No safety tuning, RLHF, or content filtering of any kind.
- Context is capped at 1,024 tokens.
Citation
@article{grt2026,
title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling},
author={MatterWave AI},
year={2026},
url={https://github.com/matterwaveai/grt}
}
License
Apache 2.0
- Downloads last month
- 451