| --- |
| license: apache-2.0 |
| tags: |
| - grt |
| - geometric-rotation-transformer |
| - causal-lm |
| - rotational |
| datasets: |
| - OptimalScale/ClimbMix |
| - wikimedia/wikipedia |
| - tiiuae/falcon-refinedweb |
| - HuggingFaceFW/fineweb-edu |
| - HuggingFaceTB/cosmopedia |
| - nvidia/Nemotron-Pretraining-Dataset-sample |
| - HuggingFaceTB/finemath |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization |
|
|
| > **Note:** `GRT-20M` (`16.7M` total parameters, `10.3M` non-embedding backbone parameters across `12` rotational blocks) was pretrained across an 8-corpus English mix using first-generation `PairAdam` for our NeurIPS submission (archived pre-fine-tune at [`MatterWaveAI/grt-20m-base`](https://huggingface.co/MatterWaveAI/grt-20m-base)). Keeping the exact same `16.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`38.63%` (`PairAdam`) → `45.03%` (`HuygensSymplecticAdam`, `+6.40%`)** and cuts `GSM8K` held-out math perplexity from **`60.12` → `9.11` (`6.60×` reduction vs. Zero-Shot, `-27.2%` vs. `PairAdam`)**. For the multi-round curriculum checkpoint (**`48.17%`** 9-task MC average, **`98.20%`** `GSM8K` verification, **`7.65`** PPL), see [`MatterWaveAI/grt-20m-curriculum`](https://huggingface.co/MatterWaveAI/grt-20m-curriculum). |
|
|
| A Geometric Rotation Transformer pretrained on a mix of eight English corpora, |
| including FineWeb-Edu, Cosmopedia, and FineMath. GRT replaces the dense linear |
| projections of a standard transformer with cascades of learned Givens rotations |
| on SO(D), so the residual path contains no `nn.Linear` at all. |
|
|
| **This is a research preview, not a product model.** It exists to show that the |
| architecture trains stably and evaluates end-to-end through the same OpenAI-HTTP |
| serving seam as every other model, and to give CPU-only CI a cheap, always- |
| available upstream. At this scale the output is grammatical and locally |
| coherent but not factual — see the real samples below before expecting anything |
| else. It is not instruction-tuned and has no chat template. |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | **Total parameters** | 16,730,869 (16.7M) | |
| | **Non-embedding parameters** | 10,280,869 (10.3M) | |
| | **d_model** | 256 | |
| | **n_blocks** | 12 | |
| | **n_heads** | 4 | |
| | **n_qkv_rotations** | 11 | |
| | **ffn_n_rotations** | 21 | |
| | **Vocabulary** | 50,000 (BPE) | |
| | **Max sequence length** | 1,024 | |
| | **Precision** | float32 | |
| |
| Parameter counts are measured from the published `model.safetensors`. "20m" is |
| the size-class label, not a parameter count. The embedding dominates: 6.45M of |
| 16.7M parameters are the token embedding (50,000 tokens x 128 phase angles, |
| plus a log-magnitude per token). |
| |
| ## Training |
| |
| | | | |
| |---|---| |
| | **Base Pretraining Data** | ClimbMix, Wikipedia, RefinedWeb, FineWeb-Edu, Cosmopedia, Nemotron, FineMath, StackExchange | |
| | **Base Pretraining Steps** | 158,000 | |
| | **Base Pretraining Optimizer** | PairAdam (`MatterWaveAI/grt-20m-base`) | |
| | **Base Peak LR** | 1e-3, cosine decay to 10% | |
| | **Base Training Loss** | 2.114 | |
| | **Task Adaptation Optimizer** | `HuygensSymplecticAdam` (`SymplecticAdam` two-stage curriculum) | |
| |
| ## Sample output (greedy, temperature 0) |
| |
| The three reference completions, generated by the published weights through |
| `transformers` f32, and reproduced token-for-token identically by the vLLM and |
| llama.cpp serving paths: |
| |
| | Prompt | Greedy continuation (20 tokens) | |
| |---|---| |
| | `The capital of France is` | ` the capital of the capital of the capital of the capital of the capital of the capital of the capital` | |
| | `Once upon a time` | `, there was a great idea to help us understand the complexities of the concept of the concept of the` | |
| | `In the beginning` | `, the first term is the same, but the second term is the same.\n\nThe problem` | |
| |
| This is what the model does: locally plausible, globally wrong. That is the |
| expected behaviour of a 16.7M base model at loss 2.114, not a serving bug. |
| |
| ## Benchmarks: NeurIPS Submission Baseline (`PairAdam`) vs. New Optimizer (`HuygensSymplecticAdam`) |
| |
| Evaluated on the `SLM-Bench` small-language-model suite (`training/slm_bench.py` / `grt-slm-bench`) starting from the exact same `16.7M`-parameter (`10.3M` non-embedding) NeurIPS submission checkpoint: |
| |
| ### 1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples) |
| |
| | Task (`SLM-Bench`) | Split (`n`) | Random | `GRT-20M` Zero-Shot | NeurIPS Baseline (`PairAdam`) | `SymplecticAdam` | New Optimizer (`HuygensSymplecticAdam`) | Gain vs. `PairAdam` | |
| |---|---|---:|---:|---:|---:|---:|---:| |
| | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `50.50%` | `55.70%` | `74.00%` | **`78.20%`** | **`+22.50%`** (`+28.20%` vs. random) | |
| | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `62.17%` | `53.43%` | `62.17%` | **`62.35%`** | **`+8.92%`** (`+12.35%` vs. random) | |
| | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | `53.97%` | `53.37%` | **`53.97%`** | **`53.97%`** | **`+0.60%`** (`+3.97%` vs. random) | |
| | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `51.93%` | `52.41%` | **`52.41%`** | **`52.41%`** | **`+0.48%`** vs. zero-shot | |
| | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `34.20%` | `25.60%` | `36.00%` | **`37.20%`** | **`+11.60%`** (`+12.20%` vs. random) | |
| | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `26.43%` | `26.85%` | `34.30%` | **`36.70%`** | **`+9.85%`** (`+11.70%` vs. random) | |
| | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `28.70%` | `31.80%` | **`32.70%`** | **`32.70%`** | **`+0.90%`** (`+7.70%` vs. random) | |
| | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `25.34%` | `25.43%` | `26.79%` | **`27.13%`** | **`+1.70%`** (`+2.13%` vs. random) | |
| | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `19.16%` | `23.10%` | `23.91%` | **`24.57%`** | **`+1.47%`** (`+5.41%` vs. zero-shot) | |
| | **9-Task Multiple-Choice Macro Average** | `13,144` total | `35.56%` | `39.16%` | `38.63%` | `44.03%` | **`45.03%`** | **`+6.40%`** (`+9.47%` vs. random) | |
|
|
| ### 2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`) |
|
|
| | Task (`SLM-Bench`) | Split (`n`) | `GRT-20M` Zero-Shot | NeurIPS Baseline (`PairAdam`) | `SymplecticAdam` | New Optimizer (`HuygensSymplecticAdam`) | Improvement | |
| |---|---|---:|---:|---:|---:|---:| |
| | **GSM8K Held-Out Perplexity (`PPL` ↓)** | `test` (`500`) | `60.12` | `12.52` | `10.70` | **`9.11`** | **`6.60×` lower vs. Zero-Shot (`-27.2%` vs. `PairAdam`)** | |
| | **GSM8K 4-Way Numerical Verification (`Acc` ↑)** | `test` (`500`) | `21.00%` | `28.80%` | `91.40%` | **`93.20%`** | **`+64.40%` vs. `PairAdam`** | |
|
|
| Reproduce with `grt-slm-bench`: |
|
|
| ```bash |
| python -m training.slm_bench --checkpoint ./ --optimizer huygens_symplectic_adam \ |
| --tasks boolq,arc_easy,arc_challenge,openbookqa,piqa,winogrande,hellaswag,qnli,commonsenseqa,gsm8k \ |
| --device cuda |
| ``` |
|
|
| Zero-shot, via lm-evaluation-harness: |
|
|
| ```bash |
| lm_eval --model hf --model_args pretrained=MatterWaveAI/grt-20m \ |
| --tasks boolq,arc_easy --batch_size 64 |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| import grt # registers model_type "grt" with the transformers Auto classes |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-20m", dtype=torch.float32) |
| tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-20m") |
| |
| input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids |
| output = model.generate(input_ids, max_new_tokens=50, do_sample=False) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional |
| and load automatically when installed. |
|
|
| > **`trust_remote_code=True` is not supported for this repository.** The model |
| > code is distributed as a pip package rather than as Hub remote code, because |
| > the implementation spans a package with absolute imports that a flat Hub |
| > snapshot cannot resolve. Install `geometric-rotation-transformer` first; |
| > `import grt` then registers `model_type: "grt"` so the Auto classes resolve |
| > these weights. |
| |
| ### GGUF |
| |
| `grt-20m-f32.gguf` is the same weights in GGUF form, loadable by the MatterWave |
| llama.cpp fork. Verify any pull against `SHA256SUMS` in this repo: |
| |
| ```bash |
| sha256sum --ignore-missing -c SHA256SUMS |
| ``` |
| |
| ## Architecture |
| |
| GRT replaces the standard Q/K/V projections and FFN weight matrices with |
| cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is |
| modulated by data-dependent Clifford features: |
| |
| ``` |
| feat = [S0, V1, V2, B12] # scalar energy, direction, cross-pair bivector |
| theta = angles + tanh(modulation) # base angle + content-dependent nudge |
| r0 = x0*cos(theta) - x1*sin(theta) |
| r1 = x0*sin(theta) + x1*cos(theta) |
| ``` |
| |
| Components: |
| |
| - **RotationalEmbedding** — tokens stored as phase angles on the unit hypersphere |
| - **RotationalAttention** — Q/K/V via Givens cascades |
| - **RotationalFFN** — feed-forward via meta-coupled rotation stages |
| - **PearlNorm** — pair-radius normalization preserving per-pair phase |
| - **RadialGELU** — activation gating pair magnitudes, preserving direction |
| - **PairAdam** — AdamW with factored second moments: 1D params (angles, scales) |
| use standard diagonal moments, 2D params (coupling matrices) use a row factor |
| [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n) |
| instead of O(mn). Optionally clamps per-step angular updates. |
| |
| Because every projection is a rotation, the transform is norm-preserving by |
| construction rather than by normalization. |
| |
| ## Intended Use & Limitations |
| |
| Intended for research into structured/rotational architectures and as a |
| deterministic CI upstream. Not intended for production use, deployment, or any |
| task requiring factual output. |
| |
| Capabilities this model does not have — the platform metadata reflects this |
| (`supports_tools: false`, `supports_response_schema: false`, |
| `supports_vision: false`): |
|
|
| - **No instruction tuning.** This is a base model trained on next-token |
| prediction only. |
| - **No chat template.** There is no `chat_template` in this repo; an |
| OpenAI-compatible `/v1/chat/completions` endpoint serves it by concatenating |
| the messages, which is fine for a demo and wrong for anything else. |
| - **No tool calling, no structured/schema-constrained output, no vision.** |
| - English only. The mix includes web crawl (RefinedWeb, ClimbMix), encyclopedic |
| text, synthetic textbooks (Cosmopedia), math (FineMath) and Q&A |
| (StackExchange), so it inherits the coverage gaps and biases of all of them. |
| - Generated text is frequently incoherent past a clause or two. Do not expect |
| instruction-following, factual reliability, or coherent long-form output. |
| - No safety tuning, RLHF, or content filtering of any kind. |
| - Context is capped at 1,024 tokens. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{grt2026, |
| title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling}, |
| author={MatterWave AI}, |
| year={2026}, |
| url={https://github.com/matterwaveai/grt} |
| } |
| ``` |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
