README.md

metadata

license: apache-2.0
tags:
  - grt
  - geometric-rotation-transformer
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - circular-pooling
  - rotational
  - qwen3
datasets:
  - sentence-transformers/all-nli
  - sentence-transformers/stsb
language:
  - en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: MatterWaveAI/grt-30m

GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)

An embedding-only Geometric Rotation Transformer (GRTModel) contrastively fine-tuned from MatterWaveAI/grt-30m for semantic textual similarity and dense retrieval using planar-pair circular pooling (sentence-transformers).

Instead of dense linear projections (nn.Linear), GRT represents token embeddings in polar coordinates (phase_angles + log_magnitude) and transforms hidden states through cascades of content-modulated 2×2 Givens rotations on SO(D) interleaved with butterfly permutations. Because every hidden state is structured as D / 2 = 128 planar (x, y) coordinate pairs, circular pooling decomposes each pair into its unit direction (cos θ, sin θ) and radius r, averages the unit directions and radii across non-padding tokens, and recombines them into a 256-dimensional sentence embedding.

---

Model Details

Property	Value
Architecture	GRTModel (model_type: "grt", encoder/backbone without causal LM head)
Base model	MatterWaveAI/grt-30m
Pooling strategy	circular (Pooling(embedding_dimension=256, pooling_mode="circular") + Normalize)
Output dimensionality	256 (128 planar (x, y) pairs, L2-normalized)
Total stored parameters	29,305,497 (29.31M, lm_head.output_scale stripped)
Non-embedding backbone parameters	9,740,197 (9.74M across 12 rotational blocks)
hidden_size (d_model)	256
num_hidden_layers (n_blocks)	12
num_attention_heads (n_heads)	4 (head_dim = 64)
intermediate_size (d_ff)	1,024
n_qkv_rotations / n_mixer_rotations / ffn_n_rotations	11 / 1 / 21
Vocabulary	151,669 (Qwen 3 BPE)
Max sequence length	512 (supports up to 2,048 positions)

---

Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)

Starting from the pretrained MatterWaveAI/grt-30m causal checkpoint (lm_head.output_scale stripped to form a headless GRTModel encoder), the 151,669 lexical embedding.phase_angles (19.41M parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the 9,740,196 rotational backbone parameters + 151,669 token log_magnitude weights (9,891,865 trainable parameters) in FP32 on an Intel Arc 140T GPU (xpu:0, 32 GB) using fused oneAPI DPC++ SYCL Givens and Clifford kernels (~75s total wall-clock time):

Stage	Dataset & Budget	Objective & Hyperparameters
Parameter Freezing	embedding.phase_angles frozen (19.41M params)	9,891,865 trainable parameters (12 rotational blocks + embedding.log_magnitude); max grad norm 1.0
Stage 1: AllNLI Contrastive Alignment	sentence-transformers/all-nli (triplet split) · 150 steps (4,800 triplets, <1% of corpus)	In-batch MultipleNegativesRankingLoss + explicit hard negatives (τ = 0.05), batch_size = 32, max_length = 96, PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)
Stage 2: STS-B Rank & Metric Calibration	sentence-transformers/stsb (train split, 5,749 pairs) · 2 epochs (358 steps)	CoSENT rank-consistency loss (τ = 20.0) + 2.0 × CosineMSE, batch_size = 32, max_length = 128, PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)

---

Evaluation Results & 17-Way Classical Geometric, Symplectic & Toroidal Optimizer Ablation (STS-B)

Evaluated on the STS Benchmark (sentence-transformers/stsb) validation and test splits using circular pooling and cosine similarity across a 17-way controlled optimizer ablation on the identical 508-step schedule (150 steps AllNLI + 358 steps STS-B, seed = 42):

Rank (Test)	Optimizer Variant	Coupling LR	Stage 1 NLI Val Spearman (×100)	STS-B Val Spearman (×100)	STS-B Val Pearson (×100)	STS-B Test Spearman (×100)	STS-B Test Pearson (×100)	Optimizer State Memory (FP32)	Step Time (508 Steps)	Delta vs. PairAdam (Val / Test)
1	SymplecticAdam (Non-Squeezing Floor + Casimir Cavity)	1.0×	61.98	73.60	70.87	65.16 (BEST TEST)	65.64 (BEST TEST)	80.7 MB (2.04×)	243s ROCm	+1.81 / +2.74
2	HuygensSymplecticAdam (Huygens Splay Curriculum -> Symplectic Cavity)	1.0×	62.82 (#4 S1)	73.68 (TIED BEST VAL)	70.90	65.09 (#2 TEST, BEST COMBINED)	65.48 (#2 TEST)	80.7 MB (2.04×)	236s ROCm (FASTEST)	+1.89 / +2.67
3	PolarNaturalAdam (Polar Metric + Hessian)	1.0×	62.31	72.62	70.20	64.71 (#3 TEST)	64.94 (#4 TEST)	80.7 MB (2.04×)	4,344s XPU	+0.83 / +2.29
4	GyricAdam (Thomas–Landé Gyroscopic Projection)	1.0×	62.21	73.04	70.23	64.55 (#4 TEST)	65.03 (#3 TEST)	80.7 MB (2.04×)	420s ROCm	+1.25 / +2.13
5	HuygensAdam (Circular Order-Parameter Splay State)	1.0×	63.83 (BEST S1)	72.95	70.35	64.20 (#5 TEST)	64.69	80.7 MB (2.04×)	332s ROCm	+1.16 / +1.78
6	HessianPairAdam (Sophia-Torus, 1×)	1.0×	62.29	72.42	68.76	63.37	63.69	80.7 MB (2.04×)	2,219s XPU	+0.63 / +0.95
7	BeltramiAdam (Force-Free Helicity Vortex Cascade)	1.0×	63.37 (#3 S1)	73.63 (#3 VAL)	70.68	63.33	63.28	80.7 MB (2.04×)	321s ROCm	+1.84 / +0.91
8	HannayAdam (Adiabatic Holonomy + Polar Metric)	1.0×	61.15	72.57	69.90	63.33	63.30	80.7 MB (2.04×)	239s ROCm	+0.78 / +0.91
9	RotorAdam (Cayley Rational Retraction)	1.0×	62.67	73.22	70.99	63.29	63.09	80.7 MB (2.04×)	1,926s XPU	+1.43 / +0.87
10	ToroidalLion (Sign-Momentum on Torus)	1.0×	60.99	72.89	69.70	63.19	63.41	39.6 MB (1.00×, LOWEST)	1,791s XPU	+1.10 / +0.77
11	AdamW (Euclidean Diagonal Baseline)	1.0×	—	72.89	—	63.17	63.72	79.1 MB (2.00×)	~1,850s XPU	+1.10 / +0.75
12	PonceletAdam (Confocal Elliptic Closure Metric)	1.0×	62.62	72.75	69.87	63.15	63.74	80.7 MB (2.04×)	267s ROCm	+0.96 / +0.73
13	HessianPairAdam (Sophia-Torus, 3× Factored)	3.0×	62.72	72.78	69.17	62.88	63.16	44.9 MB (1.13×)	3,698s XPU	+0.99 / +0.46
14	CyclideAdam (Dupin Cyclide Conic + Deviatoric Twist)	1.0×	62.55	73.68 (TIED BEST VAL)	71.79 (BEST VAL)	62.87	64.76 (#5 TEST PEARSON)	80.7 MB (2.04×)	380s ROCm	+1.89 / +0.45 (+1.46 Pearson)
15	RindlerAdam (Rindler Hyperbolic Wedge Horizon)	1.0×	62.08	73.18	70.69	62.64	63.53	80.7 MB (2.04×)	272s ROCm	+1.39 / +0.22
16	PairAdam (Published Checkpoint, Factored)	3.0×	—	71.79	—	62.42	63.30	43.4 MB (1.10×)	~1,820s XPU	0.00 / 0.00
17	ToroidalMuon (Newton–Schulz Cascade)	1.0×	63.76 (#2 S1)	73.61 (#4 VAL)	71.05 (#2 VAL)	61.97	63.04	80.7 MB (2.04×)	5,775s XPU	+1.82 / -0.45
—	Pretrained GRT-30M (Zero-Shot Base)	—	45.51	45.51	44.14	28.49	29.81	0 MB	—	—

Memory, Speed & Regime Tradeoffs Across GRT Optimizers (optimization_grt.py)

For the 9,891,865 trainable parameters (39.57 MB in FP32: 190,464 rotation angles across 84 cascade tensors, 9,175,040 Cl(2,0) coupling parameters, and 526,361 radial scale / norm / log_magnitude parameters):

Optimizer	State Buffers per Param Group	Fine-Tune State Memory (9.89M trainable)	Full Pretrain State Memory (29.31M all params)	Step Speed (508 Steps)	Best Operating Regime & Tradeoff Summary
SymplecticAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + symplectic floor & cavity cascade	80.7 MB (2.04×)	388.3 MB (3.31×)	243s ROCm (1.02×)	Best Test Generalization (65.16 Test Spearman [BEST] / 65.64 Test Pearson [BEST] / 73.60 Val Spearman, +2.74 over PairAdam). Enforces Gromov–Liouville symplectic non-squeezing (c_symp = sqrt(c_base^2 + c_rms^2 * zpe_floor^2 / (u_norm^2 + zpe_floor))) so vanishing late-stage momentum cannot freeze into sharp training basins, coupled with Helmholtz–Casimir wave-cavity boundary smoothing and Cayley retraction.
HuygensSymplecticAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + curriculum splay torque & symplectic cavity	80.7 MB (2.04×)	388.3 MB (3.31×)	236s ROCm (1.00×, FASTEST)	Best Combined Val + Test Spearman (69.39 Mean Val+Test [BEST] / 73.68 Val Spearman [TIED BEST] / 65.09 Test Spearman [#2] / 65.48 Test Pearson [#2] / 62.82 Stage 1 NLI). Unifies Stage-1 Huygens circular order parameter (R_p) splay-state anti-synchronization with Stage-2 Gromov–Liouville symplectic non-squeezing and Helmholtz–Casimir wave-cavity coupling via a smooth step-and-momentum curriculum gate gamma_splay(t, m).
PolarNaturalAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + live polar radius r_p^2	80.7 MB (2.04×)	388.3 MB (3.31×)	4,344s XPU	Best Pure Riemannian Metric Generalization (64.71 Test Spearman [#2] / 64.94 Test Pearson [#3], +2.29 over PairAdam). Equalizes arc-length displacement r_p * d_theta across high- and low-energy rotation planes via the polar metric ds^2 = dr^2 + r^2 d_theta^2.
GyricAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + Thomas–Landé projection	80.7 MB (2.04×)	388.3 MB (3.31×)	420s ROCm	Best Mean (Val + Test) Balance (73.04 Val / 64.55 Test Spearman [#3] / 65.03 Test Pearson [#2]). Decomposes cascade velocities into collective orbital (L) and stage-differentiating gyric (S) components via Landé vector projection (c_lande = (L^2 - S^2) / J^2), boosting coherent rotation while retrograde Thomas precession damps incoherent inter-stage fighting.
HuygensAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + circular order parameter R_p	80.7 MB (2.04×)	388.3 MB (3.31×)	332s ROCm	Best Stage 1 NLI Zero-Shot Alignment (63.83 Stage 1 AllNLI [BEST] / 64.20 Test Spearman [#4]). Tracks the circular synchronization order parameter `R_p =
CyclideAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + conic metric & deviatoric projection	80.7 MB (2.04×)	388.3 MB (3.31×)	380s ROCm	Best Validation Rank & Pearson Correlation (73.68 Val Spearman [BEST] / 71.79 Val Pearson [BEST] / 64.76 Test Pearson). Replaces flat-torus radii with a Dupin Cyclide manifold derived from Brianchon's conic tangent theorem on the coupled pair ellipse plus a trace-free deviatoric cascade twist.
BeltramiAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + discrete stage vorticity	80.7 MB (2.04×)	388.3 MB (3.31×)	321s ROCm	High-Helicity Cascade Alignment (63.37 Stage 1 NLI / 73.63 Val Spearman [#2] / 63.33 Test Spearman). Aligns deviatoric stage updates with discrete stage vorticity (curl(u) = lambda * u) to suppress inter-stage shear turbulence.
HannayAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v) + adiabatic scale-angle connection	80.7 MB (2.04×)	388.3 MB (3.31×)	239s ROCm (1.00×)	Fastest Holonomy-Corrected Polar Optimizer (72.57 Val / 63.33 Test Spearman in 239s). Subtracts the classical Hannay adiabatic holonomy drift omega_hannay = (r1 * dr2 - r2 * dr1) / (r1^2 + r2^2) induced by simultaneous scale updates.
PonceletAdam & RindlerAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v)	80.7 MB (2.04×)	388.3 MB (3.31×)	267s–272s ROCm	Confocal Closure (PonceletAdam: 72.75 Val / 63.74 Test Pearson) & Hyperbolic Horizon Contraction (RindlerAdam: 73.18 Val / 63.53 Test Pearson). Interpolates polar/conic metrics by confocal eccentricity (PonceletAdam) or contracts the Cayley horizon under proper angular acceleration (RindlerAdam).
RotorAdam	Angles: 4× (m, v, h, c) · Non-angles: 2× (m, v)	80.7 MB (2.04×)	388.3 MB (3.31×)	1,926s XPU	Smooth Cayley Retraction Baseline (73.22 Val / 63.29 Test Spearman). Smooth rational Cayley retraction Delta_theta = -2 * r_c * arctan(lr * u / (2 * r_c)) on SO(2)^P eliminates hard clipping discontinuities.
ToroidalLion	All params: 1× (m only) + zero-state Theorem 2.1 saddle kick	39.6 MB (1.00×)	117.2 MB (1.00×)	1,791s XPU	Best Memory-Constrained Optimizer (72.89 Val / 63.19 Test Spearman). Cuts optimizer VRAM by 50% vs AdamW and 70% vs full-pretrain HessianPairAdam while beating AdamW (63.17) and PairAdam (62.42).
HessianPairAdam (1× / 3×) & PairAdam	Angles: 2×–4× · Coupling (9.18M): 1.03× (factored) or 2× (diag)	43.4–80.7 MB	198.1–388.3 MB	1,820–3,698s XPU	Foundational Toroidal Baselines (63.37 Test Spearman for HessianPairAdam 1×, 62.88 for 3× factored, 62.42 for PairAdam).
ToroidalMuon	Angles: 4× + in-place 5-step Newton–Schulz [K, P] orthogonalization	80.7 MB (2.04×)	388.3 MB (3.31×)	5,775s XPU	Stage-Orthogonalized Reference (63.76 Stage 1 AllNLI / 73.61 STS-B Val Spearman).

---

Usage

1. sentence-transformers (Recommended)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")

sentences = [
    "Plants convert sunlight into chemical energy through photosynthesis.",
    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
    "The stock market experienced volatility during the afternoon session.",
]

embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities)

2. Hugging Face transformers (GRTModel + Circular Pooling)

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-embedding"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()


def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
    radii = pairs.norm(dim=-1, keepdim=True)
    directions = pairs / torch.clamp(radii, min=1e-9)
    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)


inputs = tokenizer(
    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
    padding=True,
    truncation=True,
    return_tensors="pt",
)
with torch.no_grad():
    outputs = model(**inputs)
    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])

print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer
8. Rotational Text-to-Speech (GRTTTS): MatterWaveAI/grt-30m-tts
9. Multimodal Vision-Audio-Language-Speech (GRTOmni): MatterWaveAI/grt-30m-omni

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}