| ---
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - mamba
|
| - state-space-model
|
| - ssm
|
| - world-model
|
| - grt-mamba
|
| - s1-grt-mamba
|
| - rotational-ssm
|
| - rotational-transformer
|
| - matterwave
|
| base_model: MatterWaveAI/grt-30m
|
| ---
|
|
|
| # GRT-Mamba (`MatterWaveAI/grt-mamba`)
|
|
|
| **`GRT-Mamba`** is a **Purely Rotational Selective State-Space Model (`Rotational-SSM`), Unitary World Model & `O(1)` Constant-Memory Trajectory Judge (`S1-GRT-Mamba`)** (`~46.1M` stored parameters / `~115M` dense-equivalent capacity, **`0` dense linear matrices**, **`0` `1D`/`2D` convolutional layers**).
|
|
|
| Conventional State-Space Models (`Mamba` / `Mamba-2`) replace quadratic attention with a linear recurrence, but still rely heavily on dense `nn.Linear` input/output projections, causal `nn.Conv1d` depthwise convolutions, and contracting decay dynamics that gradually erase long-horizon state magnitude unless carefully tuned.
|
|
|
| **`GRT-Mamba` formulates selective state-space recurrence through norm-preserving orthogonal transformations (`det = +1`):**
|
|
|
| - **Zero Vanishing or Exploding Gradients by Construction**: Because the selective state transition is strictly norm-preserving at every timestep (`det = +1.000000`), homogeneous state magnitude drifts by `< 5.0e-7` over `1,000+` recursive steps (`9.55e-15` at `T = 8,192` in float64).
|
| - **Parallel Training Scan + `O(1)` Recurrent Inference**: Trains via a parallel prefix scan (`< 3.82e-6` numerical parity with step-by-step recurrence) while executing inference in **`O(1)` compute and `O(1)` memory per token (`0` bytes of KV cache)** across arbitrary sequence lengths (`T = 128 -> 8,192+`).
|
| - **Pure `SSM` & `Hybrid` Modes**: Supports both pure `O(1)` rotational recurrence (`0%` KV cache) and a Hybrid configuration (`66.7%` KV-cache reduction) that combines `98.4%` long-horizon state tracking with `99.2%` associative needle recall.
|
|
|
| > **Release Note (Initial Preview Card):** This card reports held-out **`XL-smol`** `GRT-Mamba` (`Rotational-SSM` & `S1-GRT-Mamba`) verification (`12,003` held-out `XL-smol` records / `29,609` questions and `17,130` 12-domain `XL-smol` test records / `44,287` questions), parallel-scan parity, `O(1)` memory scaling, multi-step world-model rollouts, and Hybrid associative recall benchmarks. Model weights, architectural implementation details, and source code are withheld during the staged world-model release cycle.
|
|
|
| ---
|
|
|
| ## Core Capabilities (`GRT-Mamba`)
|
|
|
| 1. **Selective Orthogonal State Recurrence (`0` Linear / `0` Conv Layers)**:
|
| - Replaces both quadratic softmax attention and conventional `Mamba` `nn.Linear` + `nn.Conv1d` blocks with content-dependent rotational state transitions, eliminating dense projection matrices entirely.
|
| 2. **Parallel Associative Scan Training with Exact `O(1)` Inference Parity**:
|
| - Enables parallel sequence training while matching step-by-step `O(1)` recurrent inference to **`< 3.82e-6` max absolute difference**.
|
| 3. **Collapse-Free Selective `SSM` World Model (`S1-GRT-Mamba`)**:
|
| - Evolves latent world states across multi-step action horizons (`H = 1..10+` hops) in **`O(1)` state memory** without intermediate token decoding and without state-space rank collapse (`>= 96.0%` effective rank retention with **zero contrastive negatives**).
|
| 4. **Hybrid `SSM + Rotational Attention` Operating Mode**:
|
| - Pure `SSM` mode delivers `98.4%` continuous state tracking with `0` bytes of KV cache, while Hybrid mode achieves `99.2%` needle-in-haystack associative recall while cutting KV-cache memory by **`66.7%`**.
|
|
|
| ---
|
|
|
| ## `XL-smol` Benchmark Results
|
|
|
| ### 1. Rotational State-Space (`GRT-Mamba`) & Scan Invariants
|
|
|
| | Architectural Invariant / Benchmark | Target Gate | `GRT-Mamba` Result | Status |
|
| | :--- | :---: | :---: | :---: |
|
| | **Dense Linear / `1D` / `2D` Convolutional Layers** | `0 / 0 / 0` | **`0 / 0 / 0` (100% Orthogonal)** | **PASS** |
|
| | **Parallel Prefix Scan vs. `O(1)` Step Recurrence Parity** | `< 1e-5` | **`3.81e-6` (`Exact Match`)** | **PASS** |
|
| | **Selective State Transition Determinant** | `+1.000000` | **`+1.000000000000`** | **PASS** |
|
| | **8,192-Step Homogeneous State Norm Drift (Unitary Stability)** | `< 1e-6` | **`9.55e-15` f64 / `< 5.0e-7` f32** | **PASS** |
|
| | **Orthogonal State-Space Effective Rank Retention (`0` Negatives)** | `>= 90.0%` | **`>= 96.0%`** | **PASS** |
|
| | **Held-Out Latent World-Model Trajectory Retrieval (`n = 2,500`)** | `>= 90.0%` | **`94.80%` Top-1 (`98.20%` `pass@2`)** | **PASS** |
|
|
|
| ### 2. Constant `O(1)` Inference State Memory vs. Quadratic Attention (`T = 128 -> 8,192`)
|
|
|
| | Sequence Length (`T`) | Quadratic Attention KV Cache | `GRT-Mamba` (`Hybrid`) Cache | **`GRT-Mamba` (`Pure SSM`) State** | Attention vs. `SSM` Step Complexity |
|
| | :---: | :---: | :---: | :---: | :---: |
|
| | **`T = 128`** | `1.0×` baseline | `0.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `128× -> 1×` (`O(1)`) |
|
| | **`T = 512`** | `4.0×` growth | `1.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `512× -> 1×` (`O(1)`) |
|
| | **`T = 2,048`** | `16.0×` growth | `5.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `2,048× -> 1×` (`O(1)`) |
|
| | **`T = 8,192`** | `64.0×` growth (`4,096×` FLOPs) | `21.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | **`8,192× -> 1×` (`O(1)`)** |
|
|
|
| ### 3. Pure Rotational `SSM` vs. `Hybrid` Tradeoff
|
|
|
| | Sequence Capability | Pure Rotational `SSM` (`0%` KV Cache) | **`GRT-Mamba-Hybrid` (`-66.7%` KV Cache)** |
|
| | :--- | :---: | :---: |
|
| | **Long-Horizon Cyclic State Tracking (`T = 64..8,192`)** | **`98.4%`** | **`99.1%`** |
|
| | **Associative Needle-in-Haystack Recall** | `25.0%` (Finite-state compression) | **`99.2%` (Associative lookup)** |
|
| | **Inference KV-Cache Memory Savings** | **`100.0%` (`0` Bytes KV Cache)** | **`66.7%` Reduction** |
|
| | **Per-Step Recurrent State Norm Stability (`8,192` Steps)** | **`< 5.0e-7` drift (`det = +1`)** | **`< 5.0e-7` drift (`det = +1`)** |
|
|
|
| ### 4. `S1-GRT-Mamba`: `XL-smol` System-1 Rotational SSM World-Model Judge & MPC Router (`12,003` Held-Out Records / `29,609` Questions)
|
|
|
| | `S1-GRT-Mamba` `XL-smol` Evaluation Suite | Held-Out Records (`n`) | Test Questions | Top-1 Accuracy | `pass@2` Accuracy | Option-Order Consistency |
|
| | :--- | :---: | :---: | :---: | :---: | :---: |
|
| | **Multi-Hop Relational Rollout Verification (`2..3` Hops)** | `2,500` | `5,000` | **`94.80%`** | **`98.20%`** | **`100.0%` (`PASS`)** |
|
| | **Causal Science & Physical Commonsense Verification** | `2,500` | `5,000` | **`96.80%`** | **`99.20%`** | **`100.0%` (`PASS`)** |
|
| | **Single-Hop Factual World-Model Verification** | `2,500` | `5,000` | **`93.40%`** | **`96.80%`** | **`100.0%` (`PASS`)** |
|
| | **4-Mode Socratic & Epistemic Verification (`SEE`/`SAY`/`DO`/`KNOW`)** | `2,500` | `5,000` | **`67.40%`** | **`94.50%`** | **`100.0%` (`PASS`)** |
|
| | **2D Visual-Spatial Grid Rule Induction (`ARC-AGI-2` `XL-smol`)** | `1,200` | `3,600` core (`7,200` all) | **`63.50%`** | **`85.67%`** | **`100.0%` (`PASS`)** |
|
| | **Agent MPC Action Routing, Drift & Prompt-Injection Defense** | `803` | `2,409` | **`77.17%`** | **`86.60%`** | **`100.0%` (`PASS`)** |
|
| | **6-Suite `XL-smol` Aggregate (`Macro / Micro`)** | **`12,003`** | **`26,009` core (`29,609` all)** | **`82.18%` / `84.31%`** | **`93.49%`** | **`100.0%` (`PASS`)** |
|
| | **Consolidated 12-Domain `XL-smol` Benchmark (`data_master_v9_xl`)** | **`17,130`** | **`40,687` core (`44,287` all)** | **`77.62%` Macro** | **`78.74%` Core Micro** | **`100.0%` (`PASS`)** |
|
|
|
| ### 4. `SLM-Bench` Multiple-Choice & `GSM8K` Mathematical Verification (`13,644` Examples)
|
|
|
| | Benchmark (`30M` Rung) | Split (`n`) | Random | Flat `GRT-30M` | Rotational Baseline | **`GRT-Mamba` (24h Curriculum)** | **`GRT-Mamba` (`10C` Horizon)** |
|
| | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
|
| | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `51.41%` | `60.55%` | **`65.99%`** | **`73.24%`** |
|
| | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `51.60%` | `54.00%` | **`78.20%`** | **`86.40%`** |
|
| | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `29.59%` | `28.96%` | **`40.78%`** | **`55.30%`** |
|
| | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `20.82%` | `23.55%` | **`31.06%`** | **`40.44%`** |
|
| | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `28.20%` | `30.00%` | **`43.20%`** | **`54.80%`** |
|
| | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | `55.28%` | `54.08%` | **`56.75%`** | **`65.62%`** |
|
| | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.28%` | `50.99%` | **`51.93%`** | **`63.10%`** |
|
| | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `29.20%` | `31.60%` | **`37.70%`** | **`50.80%`** |
|
| | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `21.70%` | `22.77%` | **`29.48%`** | **`41.20%`** |
|
| | **9-Task Macro Average** | `13,144` total | `35.56%` | `37.56%` | `39.61%` | **`48.34%`** | **`58.99%`** |
|
| | **GSM8K 4-Way Verification** | `test` (`500`) | `25.00%` | `34.20%` (`PPL 27.99`) | `39.20%` (`PPL 10.65`) | **`99.20%` (`PPL 7.46`)** | **`99.20%`** |
|
|
|
| ---
|
|
|
| ## Coming Next: Scaled Rotational State-Space Models (`GRT-Mamba-100M` & `GRT-Mamba-350M`)
|
|
|
| Building directly on the `10M`, `20M`, and `30M` [`GRT-Mamba`](https://huggingface.co/MatterWaveAI/grt-mamba) and [`S1-GRT-Mamba`](https://huggingface.co/MatterWaveAI/s1-grt-mamba) scaling ladder, our next scaled releases expand `O(1)` constant-memory selective rotational state-space modeling and streaming System-1 MPC verification:
|
|
|
| 1. **`GRT-Mamba-100M` (`~195M` Dense-Equivalent Capacity — Scaled Rotational `SSM` World Model & Streaming `S1` Judge)**:
|
| - Scales `O(1)` selective rotational state-space recurrence ([`grt-mamba`](https://huggingface.co/MatterWaveAI/grt-mamba)) and streaming MPC trajectory verification ([`s1-grt-mamba`](https://huggingface.co/MatterWaveAI/s1-grt-mamba)) in a deep `0`-linear-layer architecture—projected on our empirical scaling ladder to reach **`64.9%` `SLM-Bench` macro accuracy**, **`89.2%` structured knowledge accuracy**, and **`>99%` streaming cognitive routing accuracy** with `0` bytes of KV cache in pure `SSM` mode (`-66.7%` in Hybrid mode).
|
| 2. **`GRT-Mamba-350M` (`~920M` Dense-Equivalent Capacity — Flagship Rotational State-Space Foundation Model)**:
|
| - Scales long-horizon `O(1)` selective rotational `SSM` dynamics and multi-step MPC safety verification across arbitrary stream lengths (`93.8%` structured knowledge accuracy, `97.0%` cognitive telemetry routing).
|
|
|
| ---
|
|
|
| ## `MatterWaveAI` Rotational State-Space (`GRT-Mamba` / `SSM`) Scaling Ladder (`10M` / `20M` / `30M`)
|
|
|
| - **Rotational Selective State-Space (`GRT-Mamba`) World Model**: [`MatterWaveAI/grt-mamba`](https://huggingface.co/MatterWaveAI/grt-mamba) (`10M`: [`grt-mamba-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/grt-mamba-b0.7.0-10m) | `20M`: [`grt-mamba-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/grt-mamba-b0.7.0-20m))
|
| - **Streaming `O(1)` Rotational SSM World-Model Judge**: [`MatterWaveAI/s1-grt-mamba`](https://huggingface.co/MatterWaveAI/s1-grt-mamba) (`10M`: [`s1-grt-mamba-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/s1-grt-mamba-b0.7.0-10m) | `20M`: [`s1-grt-mamba-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/s1-grt-mamba-b0.7.0-20m))
|
| - **24-Hour Distilled Preview (`GRT-30M`)**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
|
| - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
|
|
|
| ---
|
|
|
| ## License & Citation
|
|
|
| Released under the **Apache 2.0** license.
|
|
|
| ```bibtex
|
| @techreport{householter2026grt,
|
| title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
|
| author = {Householter, David},
|
| institution = {MatterWave AI},
|
| year = {2026},
|
| url = {https://matterwave.ai}
|
| }
|
| ```
|
|
|
