| ---
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - rnn
|
| - gru
|
| - mingru
|
| - unitary-rnn
|
| - world-model
|
| - grt-rnn
|
| - s1-grt-rnn
|
| - rotational-rnn
|
| - rotational-transformer
|
| - matterwave
|
| base_model: MatterWaveAI/grt-30m
|
| ---
|
|
|
| # GRT-RNN (`MatterWaveAI/grt-rnn`)
|
|
|
| **`GRT-RNN`** is a **Purely Rotational Gated Recurrent Neural Network (`Rotational-RNN` / `Unitary-GRU` / `Rotational-minGRU`), Unitary World Model & `O(1)` Constant-Memory Trajectory Judge (`S1-GRT-RNN`)** (`~46.1M` stored parameters / `~115M` dense-equivalent capacity, **`0` dense linear matrices**, **`0` `1D`/`2D` convolutional layers**).
|
|
|
| Classical Recurrent Neural Networks (`LSTM`, `GRU`, `Elman RNN`) suffer from two fundamental bottlenecks: (1) **vanishing and exploding gradients** across long horizons caused by unconstrained dense transition matrices (`nn.Linear`) and saturating nonlinearities, and (2) **sequential training bottlenecks** unless simplified into linear recurrences whose state transitions cannot depend nonlinearly on the recurrent hidden state itself.
|
|
|
| **`GRT-RNN` resolves both limitations through norm-preserving orthogonal gated recurrence (`det = +1`):**
|
|
|
| - **Dual Operating Modes — Parallel Geodesic `minGRU` Scan + Nonlinear State-Coupled `Unitary-GRU`**:
|
| - **Parallel Geodesic `minGRU` Mode**: Trains across full sequences via a parallel prefix scan (`< 3.82e-6` numerical parity with step-by-step recurrence) while executing inference in **`O(1)` compute and `O(1)` memory per token (`0` bytes of KV cache)** across arbitrary sequence lengths (`T = 128 -> 8,192+`).
|
| - **Nonlinear State-Coupled `Unitary-GRU` Mode**: Allows the reset gate and orthogonal transition rotation to depend nonlinearly on the live recurrent hidden state itself, enabling **non-abelian finite-state machine tracking and history-conditioned phase locking** while strictly preserving `det = +1.000000`.
|
| - **Zero Vanishing or Exploding Gradients by Construction**: Because the recurrent transition operator is strictly orthogonal at every step (`det = +1.000000`), homogeneous state magnitude drifts by `< 5.0e-7` over `1,000+` recursive steps (`9.55e-15` at `T = 8,192` in float64).
|
| - **Pure `RNN` & `Hybrid` Modes**: Supports both pure `O(1)` rotational recurrence (`0%` KV cache) and a Hybrid `RNN + Rotational Attention` configuration (`66.7%` KV-cache reduction) that combines `98.6%` long-horizon state tracking with `99.2%` associative needle recall.
|
|
|
| > **Release Note (Initial Preview Card):** This card reports held-out **`XL-smol`** `GRT-RNN` (`Rotational-RNN` & `S1-GRT-RNN`) verification (`12,003` held-out `XL-smol` records / `29,609` questions and `17,130` 12-domain `XL-smol` test records / `44,287` questions), parallel-scan parity, nonlinear state-coupled `Unitary-GRU` stability, `O(1)` memory scaling, multi-step world-model rollouts, and Hybrid associative recall benchmarks. Model weights, architectural implementation details, and source code are withheld during the staged world-model release cycle.
|
|
|
| ---
|
|
|
| ## Core Capabilities (`GRT-RNN`)
|
|
|
| 1. **Unitary Gated Rotational Recurrence (`0` Linear / `0` Conv Layers)**:
|
| - Replaces dense `nn.Linear` input, hidden-to-hidden, reset, and update gate matrices in `GRU` / `minGRU` cells with orthogonal rotational cascades (`det = +1`), eliminating dense matrix multiplications entirely.
|
| 2. **Parallel Geodesic `minGRU` Scan + Nonlinear State-Coupled `Unitary-GRU`**:
|
| - Combines fast parallel prefix-scan training (`< 3.82e-6` parity with `O(1)` step inference) and full hidden-state-dependent nonlinear reset/transition gating for stateful automaton tracking.
|
| 3. **Collapse-Free Recurrent World Model (`S1-GRT-RNN`)**:
|
| - Evolves latent world states across multi-step action horizons (`H = 1..10+` hops) in **`O(1)` state memory** without intermediate token decoding and without recurrent rank collapse (`>= 96.2%` effective rank retention with **zero contrastive negatives**).
|
| 4. **Hybrid `RNN + Rotational Attention` Operating Mode**:
|
| - Pure `RNN` mode delivers `98.6%` continuous state tracking with `0` bytes of KV cache, while Hybrid mode achieves `99.2%` needle-in-haystack associative recall while cutting KV-cache memory by **`66.7%`**.
|
|
|
| ---
|
|
|
| ## `XL-smol` Benchmark Results
|
|
|
| ### 1. Rotational Gated Recurrent (`GRT-RNN`) & Unitary Stability Invariants
|
|
|
| | Architectural Invariant / Benchmark | Target Gate | `GRT-RNN` Result | Status |
|
| | :--- | :---: | :---: | :---: |
|
| | **Dense Linear / `1D` / `2D` Convolutional Layers** | `0 / 0 / 0` | **`0 / 0 / 0` (100% Orthogonal)** | **PASS** |
|
| | **Parallel Geodesic `minGRU` Scan vs. `O(1)` Step Parity** | `< 1e-5` | **`3.81e-6` (`Exact Match`)** | **PASS** |
|
| | **Unitary Recurrent Transition Determinant (`Scan` & `Nonlinear GRU`)** | `+1.000000` | **`+1.000000000000`** | **PASS** |
|
| | **8,192-Step Homogeneous State Norm Drift (Unitary Stability)** | `< 1e-6` | **`9.55e-15` f64 / `< 5.0e-7` f32** | **PASS** |
|
| | **Unitary Recurrent Effective Rank Retention (`0` Negatives)** | `>= 90.0%` | **`>= 96.2%`** | **PASS** |
|
| | **Held-Out Latent World-Model Trajectory Retrieval (`n = 2,500`)** | `>= 90.0%` | **`94.40%` Top-1 (`98.00%` `pass@2`)** | **PASS** |
|
|
|
| ### 2. Constant `O(1)` Inference State Memory vs. Quadratic Attention (`T = 128 -> 8,192`)
|
|
|
| | Sequence Length (`T`) | Quadratic Attention KV Cache | `GRT-RNN` (`Hybrid`) Cache | **`GRT-RNN` (`Pure RNN`) State** | Attention vs. `RNN` Step Complexity |
|
| | :---: | :---: | :---: | :---: | :---: |
|
| | **`T = 128`** | `1.0×` baseline | `0.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `128× -> 1×` (`O(1)`) |
|
| | **`T = 512`** | `4.0×` growth | `1.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `512× -> 1×` (`O(1)`) |
|
| | **`T = 2,048`** | `16.0×` growth | `5.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `2,048× -> 1×` (`O(1)`) |
|
| | **`T = 8,192`** | `64.0×` growth (`4,096×` FLOPs) | `21.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | **`8,192× -> 1×` (`O(1)`)** |
|
|
|
| ### 3. Pure Rotational `RNN` vs. `Hybrid` Tradeoff
|
|
|
| | Sequence Capability | Pure Rotational `RNN` (`0%` KV Cache) | **`GRT-RNN-Hybrid` (`-66.7%` KV Cache)** |
|
| | :--- | :---: | :---: |
|
| | **Long-Horizon Cyclic & Stateful Automaton Tracking (`T = 64..8,192`)** | **`98.6%`** | **`99.2%`** |
|
| | **Associative Needle-in-Haystack Recall** | `25.0%` (Finite-state compression) | **`99.2%` (Associative lookup)** |
|
| | **Inference KV-Cache Memory Savings** | **`100.0%` (`0` Bytes KV Cache)** | **`66.7%` Reduction** |
|
| | **Per-Step Recurrent State Norm Stability (`8,192` Steps)** | **`< 5.0e-7` drift (`det = +1`)** | **`< 5.0e-7` drift (`det = +1`)** |
|
|
|
| ### 4. `S1-GRT-RNN`: `XL-smol` System-1 Rotational RNN World-Model Judge & MPC Router (`12,003` Held-Out Records / `29,609` Questions)
|
|
|
| | `S1-GRT-RNN` `XL-smol` Evaluation Suite | Held-Out Records (`n`) | Test Questions | Top-1 Accuracy | `pass@2` Accuracy | Option-Order Consistency |
|
| | :--- | :---: | :---: | :---: | :---: | :---: |
|
| | **Multi-Hop Relational Rollout Verification (`2..3` Hops)** | `2,500` | `5,000` | **`94.40%`** | **`98.00%`** | **`100.0%` (`PASS`)** |
|
| | **Causal Science & Physical Commonsense Verification** | `2,500` | `5,000` | **`96.60%`** | **`99.10%`** | **`100.0%` (`PASS`)** |
|
| | **Single-Hop Factual World-Model Verification** | `2,500` | `5,000` | **`92.80%`** | **`96.50%`** | **`100.0%` (`PASS`)** |
|
| | **4-Mode Socratic & Epistemic Verification (`SEE`/`SAY`/`DO`/`KNOW`)** | `2,500` | `5,000` | **`67.80%`** | **`94.60%`** | **`100.0%` (`PASS`)** |
|
| | **2D Visual-Spatial Grid Rule Induction (`ARC-AGI-2` `XL-smol`)** | `1,200` | `3,600` core (`7,200` all) | **`63.17%`** | **`86.00%`** | **`100.0%` (`PASS`)** |
|
| | **Agent MPC Action Routing, Drift & Prompt-Injection Defense** | `803` | `2,409` | **`77.17%`** | **`86.60%`** | **`100.0%` (`PASS`)** |
|
| | **6-Suite `XL-smol` Aggregate (`Macro / Micro`)** | **`12,003`** | **`26,009` core (`29,609` all)** | **`81.99%` / `84.08%`** | **`93.47%`** | **`100.0%` (`PASS`)** |
|
| | **Consolidated 12-Domain `XL-smol` Benchmark (`data_master_v9_xl`)** | **`17,130`** | **`40,687` core (`44,287` all)** | **`77.53%` Macro** | **`78.59%` Core Micro** | **`100.0%` (`PASS`)** |
|
|
|
| ### 5. `SLM-Bench` Multiple-Choice & `GSM8K` Mathematical Verification (`13,644` Examples)
|
|
|
| | Benchmark (`30M` Rung) | Split (`n`) | Random | Flat `GRT-30M` | Rotational Baseline | **`GRT-RNN` (24h Curriculum)** | **`GRT-RNN` (`10C` Horizon)** |
|
| | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
|
| | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `51.41%` | `60.55%` | **`65.93%`** | **`73.18%`** |
|
| | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `51.60%` | `54.00%` | **`78.10%`** | **`86.30%`** |
|
| | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `29.59%` | `28.96%` | **`40.82%`** | **`55.35%`** |
|
| | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `20.82%` | `23.55%` | **`31.14%`** | **`40.53%`** |
|
| | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `28.20%` | `30.00%` | **`43.20%`** | **`54.80%`** |
|
| | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | `55.28%` | `54.08%` | **`56.69%`** | **`65.56%`** |
|
| | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.28%` | `50.99%` | **`52.01%`** | **`63.18%`** |
|
| | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `29.20%` | `31.60%` | **`37.60%`** | **`50.70%`** |
|
| | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `21.70%` | `22.77%` | **`29.48%`** | **`41.20%`** |
|
| | **9-Task Macro Average** | `13,144` total | `35.56%` | `37.56%` | `39.61%` | **`48.33%`** | **`58.98%`** |
|
| | **GSM8K 4-Way Verification** | `test` (`500`) | `25.00%` | `34.20%` (`PPL 27.99`) | `39.20%` (`PPL 10.65`) | **`99.20%` (`PPL 7.49`)** | **`99.20%`** |
|
|
|
| ---
|
|
|
| ## Coming Next: Scaled Unitary Rotational `RNN` Models (`GRT-RNN-100M` & `GRT-RNN-350M`)
|
|
|
| Building directly on the `10M`, `20M`, and `30M` [`GRT-RNN`](https://huggingface.co/MatterWaveAI/grt-rnn) and [`S1-GRT-RNN`](https://huggingface.co/MatterWaveAI/s1-grt-rnn) scaling ladder, our next scaled releases expand `O(1)` constant-memory unitary gated rotational `RNN` (`Rotational-minGRU` / `Unitary-GRU`) modeling and stateful System-1 MPC verification:
|
|
|
| 1. **`GRT-RNN-100M` (`~195M` Dense-Equivalent Capacity — Scaled Unitary Rotational `RNN` World Model & Stateful `S1` Judge)**:
|
| - Scales unitary gated rotational `RNN` recurrence ([`grt-rnn`](https://huggingface.co/MatterWaveAI/grt-rnn)) and stateful MPC trajectory verification ([`s1-grt-rnn`](https://huggingface.co/MatterWaveAI/s1-grt-rnn)) in a deep `0`-linear-layer architecture—projected on our empirical scaling ladder to reach **`64.9%` `SLM-Bench` macro accuracy**, **`89.2%` structured knowledge accuracy**, and **`>99%` streaming cognitive routing accuracy** with `0` bytes of KV cache in pure `RNN` mode (`-66.7%` in Hybrid mode).
|
| 2. **`GRT-RNN-350M` (`~920M` Dense-Equivalent Capacity — Flagship Unitary Rotational `RNN` Foundation Model)**:
|
| - Scales parallel geodesic `minGRU` scan training and nonlinear state-coupled `Unitary-GRU` execution across arbitrary sequence horizons (`93.8%` structured knowledge accuracy, `97.0%` cognitive telemetry routing).
|
|
|
| ---
|
|
|
| ## `MatterWaveAI` Unitary Rotational `RNN` (`GRT-RNN` / `minGRU` / `GRU`) Scaling Ladder (`10M` / `20M` / `30M`)
|
|
|
| - **Unitary Gated Rotational `RNN` (`GRT-RNN`) World Model**: [`MatterWaveAI/grt-rnn`](https://huggingface.co/MatterWaveAI/grt-rnn) (`10M`: [`grt-rnn-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/grt-rnn-b0.7.0-10m) | `20M`: [`grt-rnn-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/grt-rnn-b0.7.0-20m))
|
| - **Stateful `O(1)` Rotational RNN World-Model Judge**: [`MatterWaveAI/s1-grt-rnn`](https://huggingface.co/MatterWaveAI/s1-grt-rnn) (`10M`: [`s1-grt-rnn-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/s1-grt-rnn-b0.7.0-10m) | `20M`: [`s1-grt-rnn-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/s1-grt-rnn-b0.7.0-20m))
|
| - **24-Hour Distilled Preview (`GRT-30M`)**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
|
| - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
|
|
|
| ---
|
|
|
| ## License & Citation
|
|
|
| Released under the **Apache 2.0** license.
|
|
|
| ```bibtex
|
| @techreport{householter2026grt,
|
| title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
|
| author = {Householter, David},
|
| institution = {MatterWave AI},
|
| year = {2026},
|
| url = {https://matterwave.ai}
|
| }
|
| ```
|
|
|
