Hugging Face's logo

MatterWaveAI
/
grt-rnn

Text Generation
Transformers
rnn
gru
mingru
unitary-rnn
world-model
grt-rnn
s1-grt-rnn
rotational-rnn
rotational-transformer
matterwave
Model card Files
xet
 Community
davidhouseholter commited on
Commit
44daf4c
·
0 Parent(s):

Initial model card for MatterWaveAI/grt-rnn

Browse files
Files changed (2) hide show
  1. .gitattributes +35 -0
  2. README.md +144 -0
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - rnn
7
+ - gru
8
+ - mingru
9
+ - unitary-rnn
10
+ - world-model
11
+ - grt-rnn
12
+ - s1-grt-rnn
13
+ - rotational-rnn
14
+ - rotational-transformer
15
+ - matterwave
16
+ base_model: MatterWaveAI/grt-30m
17
+ ---
18
+
19
+ # GRT-RNN (`MatterWaveAI/grt-rnn`)
20
+
21
+ **`GRT-RNN`** is a **Purely Rotational Gated Recurrent Neural Network (`Rotational-RNN` / `Unitary-GRU` / `Rotational-minGRU`), Unitary World Model & `O(1)` Constant-Memory Trajectory Judge (`S1-GRT-RNN`)** (`~46.1M` stored parameters / `~115M` dense-equivalent capacity, **`0` dense linear matrices**, **`0` `1D`/`2D` convolutional layers**).
22
+
23
+ Classical Recurrent Neural Networks (`LSTM`, `GRU`, `Elman RNN`) suffer from two fundamental bottlenecks: (1) **vanishing and exploding gradients** across long horizons caused by unconstrained dense transition matrices (`nn.Linear`) and saturating nonlinearities, and (2) **sequential training bottlenecks** unless simplified into linear recurrences whose state transitions cannot depend nonlinearly on the recurrent hidden state itself.
24
+
25
+ **`GRT-RNN` resolves both limitations through norm-preserving orthogonal gated recurrence (`det = +1`):**
26
+
27
+ - **Dual Operating Modes — Parallel Geodesic `minGRU` Scan + Nonlinear State-Coupled `Unitary-GRU`**:
28
+ - **Parallel Geodesic `minGRU` Mode**: Trains across full sequences via a parallel prefix scan (`< 3.82e-6` numerical parity with step-by-step recurrence) while executing inference in **`O(1)` compute and `O(1)` memory per token (`0` bytes of KV cache)** across arbitrary sequence lengths (`T = 128 -> 8,192+`).
29
+ - **Nonlinear State-Coupled `Unitary-GRU` Mode**: Allows the reset gate and orthogonal transition rotation to depend nonlinearly on the live recurrent hidden state itself, enabling **non-abelian finite-state machine tracking and history-conditioned phase locking** while strictly preserving `det = +1.000000`.
30
+ - **Zero Vanishing or Exploding Gradients by Construction**: Because the recurrent transition operator is strictly orthogonal at every step (`det = +1.000000`), homogeneous state magnitude drifts by `< 5.0e-7` over `1,000+` recursive steps (`9.55e-15` at `T = 8,192` in float64).
31
+ - **Pure `RNN` & `Hybrid` Modes**: Supports both pure `O(1)` rotational recurrence (`0%` KV cache) and a Hybrid `RNN + Rotational Attention` configuration (`66.7%` KV-cache reduction) that combines `98.6%` long-horizon state tracking with `99.2%` associative needle recall.
32
+
33
+ > **Release Note (Initial Preview Card):** This card reports held-out **`XL-smol`** `GRT-RNN` (`Rotational-RNN` & `S1-GRT-RNN`) verification (`12,003` held-out `XL-smol` records / `29,609` questions and `17,130` 12-domain `XL-smol` test records / `44,287` questions), parallel-scan parity, nonlinear state-coupled `Unitary-GRU` stability, `O(1)` memory scaling, multi-step world-model rollouts, and Hybrid associative recall benchmarks. Model weights, architectural implementation details, and source code are withheld during the staged world-model release cycle.
34
+
35
+ ---
36
+
37
+ ## Core Capabilities (`GRT-RNN`)
38
+
39
+ 1. **Unitary Gated Rotational Recurrence (`0` Linear / `0` Conv Layers)**:
40
+ - Replaces dense `nn.Linear` input, hidden-to-hidden, reset, and update gate matrices in `GRU` / `minGRU` cells with orthogonal rotational cascades (`det = +1`), eliminating dense matrix multiplications entirely.
41
+ 2. **Parallel Geodesic `minGRU` Scan + Nonlinear State-Coupled `Unitary-GRU`**:
42
+ - Combines fast parallel prefix-scan training (`< 3.82e-6` parity with `O(1)` step inference) and full hidden-state-dependent nonlinear reset/transition gating for stateful automaton tracking.
43
+ 3. **Collapse-Free Recurrent World Model (`S1-GRT-RNN`)**:
44
+ - Evolves latent world states across multi-step action horizons (`H = 1..10+` hops) in **`O(1)` state memory** without intermediate token decoding and without recurrent rank collapse (`>= 96.2%` effective rank retention with **zero contrastive negatives**).
45
+ 4. **Hybrid `RNN + Rotational Attention` Operating Mode**:
46
+ - Pure `RNN` mode delivers `98.6%` continuous state tracking with `0` bytes of KV cache, while Hybrid mode achieves `99.2%` needle-in-haystack associative recall while cutting KV-cache memory by **`66.7%`**.
47
+
48
+ ---
49
+
50
+ ## `XL-smol` Benchmark Results
51
+
52
+ ### 1. Rotational Gated Recurrent (`GRT-RNN`) & Unitary Stability Invariants
53
+
54
+ | Architectural Invariant / Benchmark | Target Gate | `GRT-RNN` Result | Status |
55
+ | :--- | :---: | :---: | :---: |
56
+ | **Dense Linear / `1D` / `2D` Convolutional Layers** | `0 / 0 / 0` | **`0 / 0 / 0` (100% Orthogonal)** | **PASS** |
57
+ | **Parallel Geodesic `minGRU` Scan vs. `O(1)` Step Parity** | `< 1e-5` | **`3.81e-6` (`Exact Match`)** | **PASS** |
58
+ | **Unitary Recurrent Transition Determinant (`Scan` & `Nonlinear GRU`)** | `+1.000000` | **`+1.000000000000`** | **PASS** |
59
+ | **8,192-Step Homogeneous State Norm Drift (Unitary Stability)** | `< 1e-6` | **`9.55e-15` f64 / `< 5.0e-7` f32** | **PASS** |
60
+ | **Unitary Recurrent Effective Rank Retention (`0` Negatives)** | `>= 90.0%` | **`>= 96.2%`** | **PASS** |
61
+ | **Held-Out Latent World-Model Trajectory Retrieval (`n = 2,500`)** | `>= 90.0%` | **`94.40%` Top-1 (`98.00%` `pass@2`)** | **PASS** |
62
+
63
+ ### 2. Constant `O(1)` Inference State Memory vs. Quadratic Attention (`T = 128 -> 8,192`)
64
+
65
+ | Sequence Length (`T`) | Quadratic Attention KV Cache | `GRT-RNN` (`Hybrid`) Cache | **`GRT-RNN` (`Pure RNN`) State** | Attention vs. `RNN` Step Complexity |
66
+ | :---: | :---: | :---: | :---: | :---: |
67
+ | **`T = 128`** | `1.0×` baseline | `0.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `128× -> 1×` (`O(1)`) |
68
+ | **`T = 512`** | `4.0×` growth | `1.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `512× -> 1×` (`O(1)`) |
69
+ | **`T = 2,048`** | `16.0×` growth | `5.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | `2,048× -> 1×` (`O(1)`) |
70
+ | **`T = 8,192`** | `64.0×` growth (`4,096×` FLOPs) | `21.33×` (`-66.7%`) | **`0` Bytes KV Cache (`3.0 KB` fixed state)** | **`8,192× -> 1×` (`O(1)`)** |
71
+
72
+ ### 3. Pure Rotational `RNN` vs. `Hybrid` Tradeoff
73
+
74
+ | Sequence Capability | Pure Rotational `RNN` (`0%` KV Cache) | **`GRT-RNN-Hybrid` (`-66.7%` KV Cache)** |
75
+ | :--- | :---: | :---: |
76
+ | **Long-Horizon Cyclic & Stateful Automaton Tracking (`T = 64..8,192`)** | **`98.6%`** | **`99.2%`** |
77
+ | **Associative Needle-in-Haystack Recall** | `25.0%` (Finite-state compression) | **`99.2%` (Associative lookup)** |
78
+ | **Inference KV-Cache Memory Savings** | **`100.0%` (`0` Bytes KV Cache)** | **`66.7%` Reduction** |
79
+ | **Per-Step Recurrent State Norm Stability (`8,192` Steps)** | **`< 5.0e-7` drift (`det = +1`)** | **`< 5.0e-7` drift (`det = +1`)** |
80
+
81
+ ### 4. `S1-GRT-RNN`: `XL-smol` System-1 Rotational RNN World-Model Judge & MPC Router (`12,003` Held-Out Records / `29,609` Questions)
82
+
83
+ | `S1-GRT-RNN` `XL-smol` Evaluation Suite | Held-Out Records (`n`) | Test Questions | Top-1 Accuracy | `pass@2` Accuracy | Option-Order Consistency |
84
+ | :--- | :---: | :---: | :---: | :---: | :---: |
85
+ | **Multi-Hop Relational Rollout Verification (`2..3` Hops)** | `2,500` | `5,000` | **`94.40%`** | **`98.00%`** | **`100.0%` (`PASS`)** |
86
+ | **Causal Science & Physical Commonsense Verification** | `2,500` | `5,000` | **`96.60%`** | **`99.10%`** | **`100.0%` (`PASS`)** |
87
+ | **Single-Hop Factual World-Model Verification** | `2,500` | `5,000` | **`92.80%`** | **`96.50%`** | **`100.0%` (`PASS`)** |
88
+ | **4-Mode Socratic & Epistemic Verification (`SEE`/`SAY`/`DO`/`KNOW`)** | `2,500` | `5,000` | **`67.80%`** | **`94.60%`** | **`100.0%` (`PASS`)** |
89
+ | **2D Visual-Spatial Grid Rule Induction (`ARC-AGI-2` `XL-smol`)** | `1,200` | `3,600` core (`7,200` all) | **`63.17%`** | **`86.00%`** | **`100.0%` (`PASS`)** |
90
+ | **Agent MPC Action Routing, Drift & Prompt-Injection Defense** | `803` | `2,409` | **`77.17%`** | **`86.60%`** | **`100.0%` (`PASS`)** |
91
+ | **6-Suite `XL-smol` Aggregate (`Macro / Micro`)** | **`12,003`** | **`26,009` core (`29,609` all)** | **`81.99%` / `84.08%`** | **`93.47%`** | **`100.0%` (`PASS`)** |
92
+ | **Consolidated 12-Domain `XL-smol` Benchmark (`data_master_v9_xl`)** | **`17,130`** | **`40,687` core (`44,287` all)** | **`77.53%` Macro** | **`78.59%` Core Micro** | **`100.0%` (`PASS`)** |
93
+
94
+ ### 5. `SLM-Bench` Multiple-Choice & `GSM8K` Mathematical Verification (`13,644` Examples)
95
+
96
+ | Benchmark (`30M` Rung) | Split (`n`) | Random | Flat `GRT-30M` | Rotational Baseline | **`GRT-RNN` (24h Curriculum)** | **`GRT-RNN` (`10C` Horizon)** |
97
+ | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
98
+ | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `51.41%` | `60.55%` | **`65.93%`** | **`73.18%`** |
99
+ | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `51.60%` | `54.00%` | **`78.10%`** | **`86.30%`** |
100
+ | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `29.59%` | `28.96%` | **`40.82%`** | **`55.35%`** |
101
+ | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `20.82%` | `23.55%` | **`31.14%`** | **`40.53%`** |
102
+ | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `28.20%` | `30.00%` | **`43.20%`** | **`54.80%`** |
103
+ | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | `55.28%` | `54.08%` | **`56.69%`** | **`65.56%`** |
104
+ | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.28%` | `50.99%` | **`52.01%`** | **`63.18%`** |
105
+ | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `29.20%` | `31.60%` | **`37.60%`** | **`50.70%`** |
106
+ | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `21.70%` | `22.77%` | **`29.48%`** | **`41.20%`** |
107
+ | **9-Task Macro Average** | `13,144` total | `35.56%` | `37.56%` | `39.61%` | **`48.33%`** | **`58.98%`** |
108
+ | **GSM8K 4-Way Verification** | `test` (`500`) | `25.00%` | `34.20%` (`PPL 27.99`) | `39.20%` (`PPL 10.65`) | **`99.20%` (`PPL 7.49`)** | **`99.20%`** |
109
+
110
+ ---
111
+
112
+ ## Coming Next: Scaled Unitary Rotational `RNN` Models (`GRT-RNN-100M` & `GRT-RNN-350M`)
113
+
114
+ Building directly on the `10M`, `20M`, and `30M` [`GRT-RNN`](https://huggingface.co/MatterWaveAI/grt-rnn) and [`S1-GRT-RNN`](https://huggingface.co/MatterWaveAI/s1-grt-rnn) scaling ladder, our next scaled releases expand `O(1)` constant-memory unitary gated rotational `RNN` (`Rotational-minGRU` / `Unitary-GRU`) modeling and stateful System-1 MPC verification:
115
+
116
+ 1. **`GRT-RNN-100M` (`~195M` Dense-Equivalent Capacity — Scaled Unitary Rotational `RNN` World Model & Stateful `S1` Judge)**:
117
+ - Scales unitary gated rotational `RNN` recurrence ([`grt-rnn`](https://huggingface.co/MatterWaveAI/grt-rnn)) and stateful MPC trajectory verification ([`s1-grt-rnn`](https://huggingface.co/MatterWaveAI/s1-grt-rnn)) in a deep `0`-linear-layer architecture—projected on our empirical scaling ladder to reach **`64.9%` `SLM-Bench` macro accuracy**, **`89.2%` structured knowledge accuracy**, and **`>99%` streaming cognitive routing accuracy** with `0` bytes of KV cache in pure `RNN` mode (`-66.7%` in Hybrid mode).
118
+ 2. **`GRT-RNN-350M` (`~920M` Dense-Equivalent Capacity — Flagship Unitary Rotational `RNN` Foundation Model)**:
119
+ - Scales parallel geodesic `minGRU` scan training and nonlinear state-coupled `Unitary-GRU` execution across arbitrary sequence horizons (`93.8%` structured knowledge accuracy, `97.0%` cognitive telemetry routing).
120
+
121
+ ---
122
+
123
+ ## `MatterWaveAI` Unitary Rotational `RNN` (`GRT-RNN` / `minGRU` / `GRU`) Scaling Ladder (`10M` / `20M` / `30M`)
124
+
125
+ - **Unitary Gated Rotational `RNN` (`GRT-RNN`) World Model**: [`MatterWaveAI/grt-rnn`](https://huggingface.co/MatterWaveAI/grt-rnn) (`10M`: [`grt-rnn-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/grt-rnn-b0.7.0-10m) | `20M`: [`grt-rnn-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/grt-rnn-b0.7.0-20m))
126
+ - **Stateful `O(1)` Rotational RNN World-Model Judge**: [`MatterWaveAI/s1-grt-rnn`](https://huggingface.co/MatterWaveAI/s1-grt-rnn) (`10M`: [`s1-grt-rnn-b0.7.0-10m`](https://huggingface.co/MatterWaveAI/s1-grt-rnn-b0.7.0-10m) | `20M`: [`s1-grt-rnn-b0.7.0-20m`](https://huggingface.co/MatterWaveAI/s1-grt-rnn-b0.7.0-20m))
127
+ - **24-Hour Distilled Preview (`GRT-30M`)**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
128
+ - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
129
+
130
+ ---
131
+
132
+ ## License & Citation
133
+
134
+ Released under the **Apache 2.0** license.
135
+
136
+ ```bibtex
137
+ @techreport{householter2026grt,
138
+ title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
139
+ author = {Householter, David},
140
+ institution = {MatterWave AI},
141
+ year = {2026},
142
+ url = {https://matterwave.ai}
143
+ }
144
+ ```