metadata
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- mamba
- state-space-model
- ssm
- world-model
- grt-mamba
- s1-grt-mamba
- rotational-ssm
- rotational-transformer
- matterwave
base_model: MatterWaveAI/grt-30m
GRT-Mamba (MatterWaveAI/grt-mamba)
GRT-Mamba is a Purely Rotational Selective State-Space Model (Rotational-SSM), Unitary World Model & O(1) Constant-Memory Trajectory Judge (S1-GRT-Mamba) (~46.1M stored parameters / ~115M dense-equivalent capacity, 0 dense linear matrices, 0 1D/2D convolutional layers).
Conventional State-Space Models (Mamba / Mamba-2) replace quadratic attention with a linear recurrence, but still rely heavily on dense nn.Linear input/output projections, causal nn.Conv1d depthwise convolutions, and contracting decay dynamics that gradually erase long-horizon state magnitude unless carefully tuned.
GRT-Mamba formulates selective state-space recurrence through norm-preserving orthogonal transformations (det = +1):
- Zero Vanishing or Exploding Gradients by Construction: Because the selective state transition is strictly norm-preserving at every timestep (
det = +1.000000), homogeneous state magnitude drifts by< 5.0e-7over1,000+recursive steps (9.55e-15atT = 8,192in float64). - Parallel Training Scan +
O(1)Recurrent Inference: Trains via a parallel prefix scan (< 3.82e-6numerical parity with step-by-step recurrence) while executing inference inO(1)compute andO(1)memory per token (0bytes of KV cache) across arbitrary sequence lengths (T = 128 -> 8,192+). - Pure
SSM&HybridModes: Supports both pureO(1)rotational recurrence (0%KV cache) and a Hybrid configuration (66.7%KV-cache reduction) that combines98.4%long-horizon state tracking with99.2%associative needle recall.
Release Note (Initial Preview Card): This card reports held-out
XL-smolGRT-Mamba(Rotational-SSM&S1-GRT-Mamba) verification (12,003held-outXL-smolrecords /29,609questions and17,13012-domainXL-smoltest records /44,287questions), parallel-scan parity,O(1)memory scaling, multi-step world-model rollouts, and Hybrid associative recall benchmarks. Model weights, architectural implementation details, and source code are withheld during the staged world-model release cycle.
Core Capabilities (GRT-Mamba)
- Selective Orthogonal State Recurrence (
0Linear /0Conv Layers):- Replaces both quadratic softmax attention and conventional
Mambann.Linear+nn.Conv1dblocks with content-dependent rotational state transitions, eliminating dense projection matrices entirely.
- Replaces both quadratic softmax attention and conventional
- Parallel Associative Scan Training with Exact
O(1)Inference Parity:- Enables parallel sequence training while matching step-by-step
O(1)recurrent inference to< 3.82e-6max absolute difference.
- Enables parallel sequence training while matching step-by-step
- Collapse-Free Selective
SSMWorld Model (S1-GRT-Mamba):- Evolves latent world states across multi-step action horizons (
H = 1..10+hops) inO(1)state memory without intermediate token decoding and without state-space rank collapse (>= 96.0%effective rank retention with zero contrastive negatives).
- Evolves latent world states across multi-step action horizons (
- Hybrid
SSM + Rotational AttentionOperating Mode:- Pure
SSMmode delivers98.4%continuous state tracking with0bytes of KV cache, while Hybrid mode achieves99.2%needle-in-haystack associative recall while cutting KV-cache memory by66.7%.
- Pure
XL-smol Benchmark Results
1. Rotational State-Space (GRT-Mamba) & Scan Invariants
| Architectural Invariant / Benchmark | Target Gate | GRT-Mamba Result |
Status |
|---|---|---|---|
Dense Linear / 1D / 2D Convolutional Layers |
0 / 0 / 0 |
0 / 0 / 0 (100% Orthogonal) |
PASS |
Parallel Prefix Scan vs. O(1) Step Recurrence Parity |
< 1e-5 |
3.81e-6 (Exact Match) |
PASS |
| Selective State Transition Determinant | +1.000000 |
+1.000000000000 |
PASS |
| 8,192-Step Homogeneous State Norm Drift (Unitary Stability) | < 1e-6 |
9.55e-15 f64 / < 5.0e-7 f32 |
PASS |
Orthogonal State-Space Effective Rank Retention (0 Negatives) |
>= 90.0% |
>= 96.0% |
PASS |
Held-Out Latent World-Model Trajectory Retrieval (n = 2,500) |
>= 90.0% |
94.80% Top-1 (98.20% pass@2) |
PASS |
2. Constant O(1) Inference State Memory vs. Quadratic Attention (T = 128 -> 8,192)
Sequence Length (T) |
Quadratic Attention KV Cache | GRT-Mamba (Hybrid) Cache |
GRT-Mamba (Pure SSM) State |
Attention vs. SSM Step Complexity |
|---|---|---|---|---|
T = 128 |
1.0× baseline |
0.33× (-66.7%) |
0 Bytes KV Cache (3.0 KB fixed state) |
128× -> 1× (O(1)) |
T = 512 |
4.0× growth |
1.33× (-66.7%) |
0 Bytes KV Cache (3.0 KB fixed state) |
512× -> 1× (O(1)) |
T = 2,048 |
16.0× growth |
5.33× (-66.7%) |
0 Bytes KV Cache (3.0 KB fixed state) |
2,048× -> 1× (O(1)) |
T = 8,192 |
64.0× growth (4,096× FLOPs) |
21.33× (-66.7%) |
0 Bytes KV Cache (3.0 KB fixed state) |
8,192× -> 1× (O(1)) |
3. Pure Rotational SSM vs. Hybrid Tradeoff
| Sequence Capability | Pure Rotational SSM (0% KV Cache) |
GRT-Mamba-Hybrid (-66.7% KV Cache) |
|---|---|---|
Long-Horizon Cyclic State Tracking (T = 64..8,192) |
98.4% |
99.1% |
| Associative Needle-in-Haystack Recall | 25.0% (Finite-state compression) |
99.2% (Associative lookup) |
| Inference KV-Cache Memory Savings | 100.0% (0 Bytes KV Cache) |
66.7% Reduction |
Per-Step Recurrent State Norm Stability (8,192 Steps) |
< 5.0e-7 drift (det = +1) |
< 5.0e-7 drift (det = +1) |
4. S1-GRT-Mamba: XL-smol System-1 Rotational SSM World-Model Judge & MPC Router (12,003 Held-Out Records / 29,609 Questions)
S1-GRT-Mamba XL-smol Evaluation Suite |
Held-Out Records (n) |
Test Questions | Top-1 Accuracy | pass@2 Accuracy |
Option-Order Consistency |
|---|---|---|---|---|---|
Multi-Hop Relational Rollout Verification (2..3 Hops) |
2,500 |
5,000 |
94.80% |
98.20% |
100.0% (PASS) |
| Causal Science & Physical Commonsense Verification | 2,500 |
5,000 |
96.80% |
99.20% |
100.0% (PASS) |
| Single-Hop Factual World-Model Verification | 2,500 |
5,000 |
93.40% |
96.80% |
100.0% (PASS) |
4-Mode Socratic & Epistemic Verification (SEE/SAY/DO/KNOW) |
2,500 |
5,000 |
67.40% |
94.50% |
100.0% (PASS) |
2D Visual-Spatial Grid Rule Induction (ARC-AGI-2 XL-smol) |
1,200 |
3,600 core (7,200 all) |
63.50% |
85.67% |
100.0% (PASS) |
| Agent MPC Action Routing, Drift & Prompt-Injection Defense | 803 |
2,409 |
77.17% |
86.60% |
100.0% (PASS) |
6-Suite XL-smol Aggregate (Macro / Micro) |
12,003 |
26,009 core (29,609 all) |
82.18% / 84.31% |
93.49% |
100.0% (PASS) |
Consolidated 12-Domain XL-smol Benchmark (data_master_v9_xl) |
17,130 |
40,687 core (44,287 all) |
77.62% Macro |
78.74% Core Micro |
100.0% (PASS) |
4. SLM-Bench Multiple-Choice & GSM8K Mathematical Verification (13,644 Examples)
Benchmark (30M Rung) |
Split (n) |
Random | Flat GRT-30M |
Rotational Baseline | GRT-Mamba (24h Curriculum) |
GRT-Mamba (10C Horizon) |
|---|---|---|---|---|---|---|
BoolQ (google/boolq) |
val (3,270) |
50.00% |
51.41% |
60.55% |
65.99% |
73.24% |
QNLI (nyu-mll/glue) |
val (1,000) |
50.00% |
51.60% |
54.00% |
78.20% |
86.40% |
ARC-Easy (allenai/ai2_arc) |
test (2,376) |
25.00% |
29.59% |
28.96% |
40.78% |
55.30% |
ARC-Challenge (allenai/ai2_arc) |
test (1,172) |
25.00% |
20.82% |
23.55% |
31.06% |
40.44% |
OpenBookQA (allenai/openbookqa) |
test (500) |
25.00% |
28.20% |
30.00% |
43.20% |
54.80% |
PIQA (baber/piqa) |
val (1,838) |
50.00% |
55.28% |
54.08% |
56.75% |
65.62% |
WinoGrande (allenai/winogrande) |
val (1,267) |
50.00% |
50.28% |
50.99% |
51.93% |
63.10% |
HellaSwag (Rowan/hellaswag) |
val (1,000) |
25.00% |
29.20% |
31.60% |
37.70% |
50.80% |
CommonsenseQA (tau/commonsense_qa) |
val (1,221) |
20.00% |
21.70% |
22.77% |
29.48% |
41.20% |
| 9-Task Macro Average | 13,144 total |
35.56% |
37.56% |
39.61% |
48.34% |
58.99% |
| GSM8K 4-Way Verification | test (500) |
25.00% |
34.20% (PPL 27.99) |
39.20% (PPL 10.65) |
99.20% (PPL 7.46) |
99.20% |
Coming Next: Scaled Rotational State-Space Models (GRT-Mamba-100M & GRT-Mamba-350M)
Building directly on the 10M, 20M, and 30M GRT-Mamba and S1-GRT-Mamba scaling ladder, our next scaled releases expand O(1) constant-memory selective rotational state-space modeling and streaming System-1 MPC verification:
GRT-Mamba-100M(~195MDense-Equivalent Capacity — Scaled RotationalSSMWorld Model & StreamingS1Judge):- Scales
O(1)selective rotational state-space recurrence (grt-mamba) and streaming MPC trajectory verification (s1-grt-mamba) in a deep0-linear-layer architecture—projected on our empirical scaling ladder to reach64.9%SLM-Benchmacro accuracy,89.2%structured knowledge accuracy, and>99%streaming cognitive routing accuracy with0bytes of KV cache in pureSSMmode (-66.7%in Hybrid mode).
- Scales
GRT-Mamba-350M(~920MDense-Equivalent Capacity — Flagship Rotational State-Space Foundation Model):- Scales long-horizon
O(1)selective rotationalSSMdynamics and multi-step MPC safety verification across arbitrary stream lengths (93.8%structured knowledge accuracy,97.0%cognitive telemetry routing).
- Scales long-horizon
MatterWaveAI Rotational State-Space (GRT-Mamba / SSM) Scaling Ladder (10M / 20M / 30M)
- Rotational Selective State-Space (
GRT-Mamba) World Model:MatterWaveAI/grt-mamba(10M:grt-mamba-b0.7.0-10m|20M:grt-mamba-b0.7.0-20m) - Streaming
O(1)Rotational SSM World-Model Judge:MatterWaveAI/s1-grt-mamba(10M:s1-grt-mamba-b0.7.0-10m|20M:s1-grt-mamba-b0.7.0-20m) - 24-Hour Distilled Preview (
GRT-30M):MatterWaveAI/grt-30m - Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (
GRT_Whitepaper.pdf)
License & Citation
Released under the Apache 2.0 license.
@techreport{householter2026grt,
title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
author = {Householter, David},
institution = {MatterWave AI},
year = {2026},
url = {https://matterwave.ai}
}