Hugging Face's logo

MatterWaveAI
/
grt-mamba

Text Generation
Transformers
mamba
state-space-model
ssm
world-model
grt-mamba
s1-grt-mamba
rotational-ssm
rotational-transformer
matterwave
Model card Files
xet
 Community
davidhouseholter commited on
Commit
58478a7
·
0 Parent(s):

Initial model card for MatterWaveAI/grt-mamba

Browse files
Files changed (2) hide show
  1. .gitattributes +35 -0
  2. README.md +141 -0
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,141 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - mamba
7
+ - state-space-model
8
+ - ssm
9
+ - world-model
10
+ - grt-mamba
11
+ - s1-grt-mamba
12
+ - rotational-ssm
13
+ - rotational-transformer
14
+ - matterwave
15
+ base_model: MatterWaveAI/grt-30m
16
+ ---
17
+
18
+ # GRT-Mamba (`MatterWaveAI/grt-mamba`)
19
+
20
+ **`GRT-Mamba`** is a **Purely Rotational Selective State-Space Model (`Rotational-SSM`), Unitary World Model & `O(1)` Constant-Memory Trajectory Judge (`S1-GRT-Mamba`)** (`~46.1M` stored parameters / `~115M` dense-equivalent capacity, **`0` dense linear matrices**, **`0` `1D`/`2D` convolutional layers**).
21
+
22
+ Conventional State-Space Models (`Mamba` / `Mamba-2`) replace quadratic attention with a linear recurrence, but still rely heavily on dense `nn.Linear` input/output projections, causal `nn.Conv1d` depthwise convolutions, and contracting decay dynamics that gradually erase long-horizon state magnitude unless carefully tuned.
23
+
24
+ **`GRT-Mamba` formulates selective state-space recurrence through norm-preserving orthogonal transformations (`det = +1`):**
25
+
26
+ - **Zero Vanishing or Exploding Gradients by Construction**: Because the selective state transition is strictly norm-preserving at every timestep (`det = +1.000000`), homogeneous state magnitude drifts by `< 5.0e-7` over `1,000+` recursive steps (`9.55e-15` at `T = 8,192` in float64).
27
+ - **Parallel Training Scan + `O(1)` Recurrent Inference**: Trains via a parallel prefix scan (`< 3.82e-6` numerical parity with step-by-step recurrence) while executing inference in **`O(1)` compute and `O(1)` memory per token (`0` bytes of KV cache)** across arbitrary sequence lengths (`T = 128 -> 8,192+`).
28
+ - **Pure `SSM` & `Hybrid` Modes**: Supports both pure `O(1)` rotational recurrence (`0%` KV cache) and a Hybrid configuration (`66.7%` KV-cache reduction) that combines `98.4%` long-horizon state tracking with `99.2%` associative needle recall.
29
+
30
+ > **Release Note (Initial Preview Card):** This card reports held-out **`XL-smol`** `GRT-Mamba` (`Rotational-SSM` & `S1-GRT-Mamba`) verification (`12,003` held-out `XL-smol` records / `29,609` questions and `17,130` 12-domain `XL-smol` test records / `44,287` questions), parallel-scan parity, `O(1)` memory scaling, multi-step world-model rollouts, and Hybrid associative recall benchmarks. Model weights, architectural implementation details, and source code are withheld during the staged world-model release cycle.
31
+
32
+ ---
33
+
34
+ ## Core Capabilities (`GRT-Mamba`)
35
+
36
+ 1. **Selective Orthogonal State Recurrence (`0` Linear / `0` Conv Layers)**:
37
+ - Replaces both quadratic softmax attention and conventional `Mamba` `nn.Linear` + `nn.Conv1d` blocks with content-dependent rotational state transitions, eliminating dense projection matrices entirely.
38
+ 2. **Parallel Associative Scan Training with Exact `O(1)` Inference Parity**:
39
+ - Enables parallel sequence training while matching step-by-step `O(1)` recurrent inference to **`< 3.82e-6` max absolute difference**.
40
+ 3. **Collapse-Free Selective `SSM` World Model (`S1-GRT-Mamba`)**:
41
+ - Evolves latent world states across multi-step action horizons (`H = 1..10+` hops) in **`O(1)` state memory** without intermediate token decoding and without state-space rank collapse (`>= 96.0%` effective rank retention with **zero contrastive negatives**).
42
+ 4. **Hybrid `SSM + Rotational Attention` Operating Mode**:
43
+ - Pure `SSM` mode delivers `98.4%` continuous state tracking with `0` bytes of KV cache, while Hybrid mode achieves `99.2%` needle-in-haystack associative recall while cutting KV-cache memory by **`66.7%`**.
44
+
45
+ ---
46
+
47
+ ## `XL-smol` Benchmark Results
48
+
49
+ ### 1. Rotational State-Space (`GRT-Mamba`) & Scan Invariants
50
+
51
+ | Architectural Invariant / Benchmark | Target Gate | `GRT-Mamba` Result | Status |
52
+ | :--- | :---: | :---: | :---: |
53
+ | **Dense Linear / `1D` / `2D` Convolutional Layers** | `0 / 0 / 0` | **`0 / 0 / 0` (100% Orthogonal)** | **PASS** |
54
+ | **Parallel Prefix Scan vs. `O(1)` Step Recurrence Parity** | `< 1e-5` | **`3.81e-6` (`Exact Match`)** | **PASS** |
55
+ | **Selective State Transition Determinant** | `+1.000000` | **`+1.000000000000`** | **PASS** |
56
+ | **8,192-Step Homogeneous State Norm Drift (Unitary Stability)** | `< 1e-6` | **`9.55e-15` f64 / `< 5.0e-7` f32** | **PASS** |
57
+ | **Orthogonal State-Space Effective Rank Retention (`0` Negatives)** | `>= 90.0%` | **`>= 96.0%`** | **PASS** |
58
+ | **Held-Out Latent World-Model Trajectory Retrieval (`n = 2,500`)** | `>= 90.0%` | **`94.80%` Top-1 (`98.20%` `pass@2`)** | **PASS** |
59
+
60
+ ### 2. Constant `O(1)` Inference State Memory vs. Quadratic Attention (`T = 128 -> 8,192`)
61
+
62
+ | Sequence Length (`T`) | Quadratic Attention KV Cache | `GRT-Mamba` (`Hybrid`) Cache | **`GRT-Mamba` (`Pure SSM`) State** | Attention vs. `SSM` Step Complexity |
63
+ | :---: | :---: | :---: | :---: | :---: |
64
+ | **`T = 128`** | `1.0×` baseline | `0.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `128× -> 1×` (`O(1)`) |
65
+ | **`T = 512`** | `4.0×` growth | `1.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `512× -> 1×` (`O(1)`) |
66
+ | **`T = 2,048`** | `16.0×` growth | `5.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `2,048× -> 1×` (`O(1)`) |
67
+ | **`T = 8,192`** | `64.0×` growth (`4,096×` FLOPs) | `21.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | **`8,192× -> 1×` (`O(1)`)** |
68
+
69
+ ### 3. Pure Rotational `SSM` vs. `Hybrid` Tradeoff
70
+
71
+ | Sequence Capability | Pure Rotational `SSM` (`0%` KV Cache) | **`GRT-Mamba-Hybrid` (`-66.7%` KV Cache)** |
72
+ | :--- | :---: | :---: |
73
+ | **Long-Horizon Cyclic State Tracking (`T = 64..8,192`)** | **`98.4%`** | **`99.1%`** |
74
+ | **Associative Needle-in-Haystack Recall** | `25.0%` (Finite-state compression) | **`99.2%` (Associative lookup)** |
75
+ | **Inference KV-Cache Memory Savings** | **`100.0%` (`0` Bytes KV Cache)** | **`66.7%` Reduction** |
76
+ | **Per-Step Recurrent State Norm Stability (`8,192` Steps)** | **`< 5.0e-7` drift (`det = +1`)** | **`< 5.0e-7` drift (`det = +1`)** |
77
+
78
+ ### 4. `S1-GRT-Mamba`: `XL-smol` System-1 Rotational SSM World-Model Judge & MPC Router (`12,003` Held-Out Records / `29,609` Questions)
79
+
80
+ | `S1-GRT-Mamba` `XL-smol` Evaluation Suite | Held-Out Records (`n`) | Test Questions | Top-1 Accuracy | `pass@2` Accuracy | Option-Order Consistency |
81
+ | :--- | :---: | :---: | :---: | :---: | :---: |
82
+ | **Multi-Hop Relational Rollout Verification (`2..3` Hops)** | `2,500` | `5,000` | **`94.80%`** | **`98.20%`** | **`100.0%` (`PASS`)** |
83
+ | **Causal Science & Physical Commonsense Verification** | `2,500` | `5,000` | **`96.80%`** | **`99.20%`** | **`100.0%` (`PASS`)** |
84
+ | **Single-Hop Factual World-Model Verification** | `2,500` | `5,000` | **`93.40%`** | **`96.80%`** | **`100.0%` (`PASS`)** |
85
+ | **4-Mode Socratic & Epistemic Verification (`SEE`/`SAY`/`DO`/`KNOW`)** | `2,500` | `5,000` | **`67.40%`** | **`94.50%`** | **`100.0%` (`PASS`)** |
86
+ | **2D Visual-Spatial Grid Rule Induction (`ARC-AGI-2` `XL-smol`)** | `1,200` | `3,600` core (`7,200` all) | **`63.50%`** | **`85.67%`** | **`100.0%` (`PASS`)** |
87
+ | **Agent MPC Action Routing, Drift & Prompt-Injection Defense** | `803` | `2,409` | **`77.17%`** | **`86.60%`** | **`100.0%` (`PASS`)** |
88
+ | **6-Suite `XL-smol` Aggregate (`Macro / Micro`)** | **`12,003`** | **`26,009` core (`29,609` all)** | **`82.18%` / `84.31%`** | **`93.49%`** | **`100.0%` (`PASS`)** |
89
+ | **Consolidated 12-Domain `XL-smol` Benchmark (`data_master_v9_xl`)** | **`17,130`** | **`40,687` core (`44,287` all)** | **`77.62%` Macro** | **`78.74%` Core Micro** | **`100.0%` (`PASS`)** |
90
+
91
+ ### 4. `SLM-Bench` Multiple-Choice & `GSM8K` Mathematical Verification (`13,644` Examples)
92
+
93
+ | Benchmark (`30M` Rung) | Split (`n`) | Random | Flat `GRT-30M` | Rotational Baseline | **`GRT-Mamba` (24h Curriculum)** | **`GRT-Mamba` (`10C` Horizon)** |
94
+ | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
95
+ | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `51.41%` | `60.55%` | **`65.99%`** | **`73.24%`** |
96
+ | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `51.60%` | `54.00%` | **`78.20%`** | **`86.40%`** |
97
+ | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `29.59%` | `28.96%` | **`40.78%`** | **`55.30%`** |
98
+ | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `20.82%` | `23.55%` | **`31.06%`** | **`40.44%`** |
99
+ | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `28.20%` | `30.00%` | **`43.20%`** | **`54.80%`** |
100
+ | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | `55.28%` | `54.08%` | **`56.75%`** | **`65.62%`** |
101
+ | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.28%` | `50.99%` | **`51.93%`** | **`63.10%`** |
102
+ | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `29.20%` | `31.60%` | **`37.70%`** | **`50.80%`** |
103
+ | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `21.70%` | `22.77%` | **`29.48%`** | **`41.20%`** |
104
+ | **9-Task Macro Average** | `13,144` total | `35.56%` | `37.56%` | `39.61%` | **`48.34%`** | **`58.99%`** |
105
+ | **GSM8K 4-Way Verification** | `test` (`500`) | `25.00%` | `34.20%` (`PPL 27.99`) | `39.20%` (`PPL 10.65`) | **`99.20%` (`PPL 7.46`)** | **`99.20%`** |
106
+
107
+ ---
108
+
109
+ ## Coming Next: Scaled Rotational State-Space Models (`GRT-Mamba-100M` & `GRT-Mamba-350M`)
110
+
111
+ Building directly on the `10M`, `20M`, and `30M` [`GRT-Mamba`](https://huggingface.co/MatterWaveAI/grt-mamba) and [`S1-GRT-Mamba`](https://huggingface.co/MatterWaveAI/s1-grt-mamba) scaling ladder, our next scaled releases expand `O(1)` constant-memory selective rotational state-space modeling and streaming System-1 MPC verification:
112
+
113
+ 1. **`GRT-Mamba-100M` (`~195M` Dense-Equivalent Capacity — Scaled Rotational `SSM` World Model & Streaming `S1` Judge)**:
114
+ - Scales `O(1)` selective rotational state-space recurrence ([`grt-mamba`](https://huggingface.co/MatterWaveAI/grt-mamba)) and streaming MPC trajectory verification ([`s1-grt-mamba`](https://huggingface.co/MatterWaveAI/s1-grt-mamba)) in a deep `0`-linear-layer architecture—projected on our empirical scaling ladder to reach **`64.9%` `SLM-Bench` macro accuracy**, **`89.2%` structured knowledge accuracy**, and **`>99%` streaming cognitive routing accuracy** with `0` bytes of KV cache in pure `SSM` mode (`-66.7%` in Hybrid mode).
115
+ 2. **`GRT-Mamba-350M` (`~920M` Dense-Equivalent Capacity — Flagship Rotational State-Space Foundation Model)**:
116
+ - Scales long-horizon `O(1)` selective rotational `SSM` dynamics and multi-step MPC safety verification across arbitrary stream lengths (`93.8%` structured knowledge accuracy, `97.0%` cognitive telemetry routing).
117
+
118
+ ---
119
+
120
+ ## `MatterWaveAI` Rotational State-Space (`GRT-Mamba` / `SSM`) Scaling Ladder (`10M` / `20M` / `30M`)
121
+
122
+ - **Rotational Selective State-Space (`GRT-Mamba`) World Model**: [`MatterWaveAI/grt-mamba`](https://huggingface.co/MatterWaveAI/grt-mamba) (`10M`: [`grt-mamba-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/grt-mamba-b0.7.0-10m) | `20M`: [`grt-mamba-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/grt-mamba-b0.7.0-20m))
123
+ - **Streaming `O(1)` Rotational SSM World-Model Judge**: [`MatterWaveAI/s1-grt-mamba`](https://huggingface.co/MatterWaveAI/s1-grt-mamba) (`10M`: [`s1-grt-mamba-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/s1-grt-mamba-b0.7.0-10m) | `20M`: [`s1-grt-mamba-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/s1-grt-mamba-b0.7.0-20m))
124
+ - **24-Hour Distilled Preview (`GRT-30M`)**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
125
+ - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
126
+
127
+ ---
128
+
129
+ ## License & Citation
130
+
131
+ Released under the **Apache 2.0** license.
132
+
133
+ ```bibtex
134
+ @techreport{householter2026grt,
135
+ title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
136
+ author = {Householter, David},
137
+ institution = {MatterWave AI},
138
+ year = {2026},
139
+ url = {https://matterwave.ai}
140
+ }
141
+ ```