| --- |
| license: apache-2.0 |
| tags: |
| - grt |
| - geometric-rotation-transformer |
| - causal-lm |
| - rotational |
| datasets: |
| - nvidia/Nemotron-Pretraining-Dataset-sample |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # GRT-10M-Base (NeurIPS Submission Pretrained Checkpoint) |
|
|
| > **Note:** This repository preserves the original, untouched `8.7M`-parameter (`2.2M` non-embedding backbone) **GRT-10M** base checkpoint trained with first-generation `PairAdam` (`336,000` steps on Nemotron) for our NeurIPS submission, prior to `SymplecticAdam` / `HuygensSymplecticAdam` curriculum fine-tuning. For the curriculum-aligned comparison, see [`MatterWaveAI/grt-10m`](https://huggingface.co/MatterWaveAI/grt-10m). |
|
|
| A Geometric Rotation Transformer pretrained on Nemotron. GRT replaces the dense |
| linear projections of a standard transformer with cascades of learned Givens |
| rotations on SO(D), so the residual path contains no `nn.Linear` at all. |
|
|
| This is a research proof-of-concept at small scale. It is published to |
| demonstrate that the architecture trains stably and evaluates end-to-end, not |
| as a capable language model. |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | **Total parameters** | 8,685,283 (8.7M) | |
| | **Non-embedding parameters** | 2,235,283 (2.2M) | |
| | **d_model** | 256 | |
| | **n_blocks** | 6 | |
| | **n_heads** | 4 | |
| | **n_qkv_rotations** | 5 | |
| | **ffn_n_rotations** | 9 | |
| | **Vocabulary** | 50,000 (BPE) | |
| | **Max sequence length** | 1,024 | |
| | **Precision** | float32 | |
| |
| The embedding dominates the parameter count: 6.45M of 8.7M parameters are the |
| token embedding (50,000 tokens x 128 phase angles, plus a log-magnitude per |
| token). |
| |
| ## Training |
| |
| | | | |
| |---|---| |
| | **Data** | Nemotron | |
| | **Steps** | 336,000 | |
| | **Optimizer** | PairAdam | |
| | **Peak LR** | 1e-3, cosine decay to 10% | |
| | **Training loss** | 2.144 | |
| |
| | Metric | Value | |
| |---|---| |
| | Validation loss | 4.033 | |
| | Validation perplexity | 56.4 | |
| | Validation accuracy | 0.400 | |
| |
| ## Benchmarks (Pre-Fine-Tune Zero-Shot & Standard `PairAdam` Baseline) |
| |
| Evaluated via `grt-slm-bench` (`training/slm_bench.py`) across 9 standard multiple-choice benchmarks (`13,144` held-out examples) and `GSM8K` held-out perplexity: |
| |
| | Task (`SLM-Bench`) | Split (`n`) | Random | `GRT-10M-Base` (Zero-Shot) | Standard `PairAdam` Fine-Tune | |
| |---|---|---:|---:|---:| |
| | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | **`62.14%`** | `61.13%` | |
| | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | **`52.70%`** | `49.70%` | |
| | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | **`52.18%`** | `50.82%` | |
| | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.51%` | **`50.75%`** | |
| | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | **`30.00%`** | `22.60%` | |
| | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | **`26.94%`** | `26.60%` | |
| | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `25.60%` | **`29.10%`** | |
| | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `22.53%` | **`24.91%`** | |
| | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `20.15%` | **`22.85%`** | |
| | **9-Task Multiple-Choice Macro Average** | `13,144` total | `35.56%` | **`38.08%`** | `37.61%` | |
|
|
| | Held-Out Math Perplexity | Split (`n`) | `GRT-10M-Base` (Zero-Shot) | Standard `PairAdam` Fine-Tune | |
| |---|---:|---:|---:| |
| | **GSM8K (`openai/gsm8k`)** | `test` (`500`) | `145.85` PPL | `37.27` PPL | |
|
|
| Reproduce with: |
|
|
| ```bash |
| python -m training.slm_bench --checkpoint ./ --tasks boolq,arc_easy --device cuda |
| ``` |
|
|
| Zero-shot, via lm-evaluation-harness: |
|
|
| ```bash |
| lm_eval --model hf --model_args pretrained=MatterWaveAI/grt-10m-base \ |
| --tasks boolq,arc_easy --batch_size 64 |
| ``` |
|
|
| ## Usage |
|
|
| ```bash |
| pip install geometric-rotation-transformer |
| ``` |
|
|
| ```python |
| from grt import GRTForCausalLM, RotationalTokenizerBPE |
| |
| model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-10m-base") |
| tokenizer = RotationalTokenizerBPE.from_pretrained("MatterWaveAI/grt-10m-base") |
| |
| input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids |
| output = model.generate(input_ids, max_new_tokens=50, temperature=0.7) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| Importing `grt` registers the architecture with the `transformers` Auto classes: |
|
|
| ```python |
| import grt # registers model_type "grt" |
| from transformers import AutoModelForCausalLM |
| |
| model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-10m-base") |
| ``` |
|
|
| The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional |
| and load automatically when installed. |
|
|
| > **`trust_remote_code=True` is not supported for this repository.** The model |
| > code is distributed as a pip package rather than as Hub remote code, because |
| > the implementation spans a package with absolute imports that a flat Hub |
| > snapshot cannot resolve. Install `geometric-rotation-transformer` first; |
| > `import grt` then registers `model_type: "grt"` so the Auto classes resolve |
| > these weights. |
| |
| ## Architecture |
| |
| GRT replaces the standard Q/K/V projections and FFN weight matrices with |
| cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is |
| modulated by data-dependent Clifford features: |
| |
| ``` |
| feat = [S0, V1, V2, B12] # scalar energy, direction, cross-pair bivector |
| theta = angles + tanh(modulation) # base angle + content-dependent nudge |
| r0 = x0*cos(theta) - x1*sin(theta) |
| r1 = x0*sin(theta) + x1*cos(theta) |
| ``` |
| |
| Components: |
| |
| - **RotationalEmbedding** — tokens stored as phase angles on the unit hypersphere |
| - **RotationalAttention** — Q/K/V via Givens cascades |
| - **RotationalFFN** — feed-forward via meta-coupled rotation stages |
| - **PearlNorm** — pair-radius normalization preserving per-pair phase |
| - **RadialGELU** — activation gating pair magnitudes, preserving direction |
| - **PairAdam** — AdamW with factored second moments: 1D params (angles, scales) |
| use standard diagonal moments, 2D params (coupling matrices) use a row factor |
| [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n) |
| instead of O(mn). Optionally clamps per-step angular updates. |
| |
| Because every projection is a rotation, the transform is norm-preserving by |
| construction rather than by normalization. |
| |
| ## Intended Use & Limitations |
| |
| Intended for research into structured/rotational architectures. Not intended for |
| production use or deployment. |
| |
| Known limitations: |
| |
| - English only, trained on a single corpus (Nemotron), so it inherits that |
| corpus's coverage gaps and biases. |
| - No instruction tuning. This is a base model trained on next-token prediction. |
| - At 8.7M parameters, generated text is frequently incoherent. Do not expect |
| instruction-following, factual reliability, or coherent long-form output. |
| - No safety tuning, RLHF, or content filtering of any kind. |
| - Context is capped at 1,024 tokens. |
| |
| ## Citation |
| |
| ```bibtex |
| @article{grt2026, |
| title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling}, |
| author={MatterWave AI}, |
| year={2026}, |
| url={https://github.com/matterwaveai/grt} |
| } |
| ``` |
| |
| ## License |
| |
| Apache 2.0 |
| |
