Hugging Face's logo

MatterWaveAI
/
grt-30m-embedding

Sentence Similarity
sentence-transformers
Safetensors
English
grt
geometric-rotation-transformer
feature-extraction
circular-pooling
rotational
qwen3
Model card Files
xet
 Community
grt-30m-embedding / README.md
davidhouseholter's picture
davidhouseholter
Update 17-way classical geometric optimizer ablation (add HuygensSymplecticAdam): README.md
00e6b36 verified
preview code
|
Raw History Blame Contribute Delete
18.7 kB
---
license: apache-2.0
tags:
- grt
- geometric-rotation-transformer
- sentence-transformers
- feature-extraction
- sentence-similarity
- circular-pooling
- rotational
- qwen3
datasets:
- sentence-transformers/all-nli
- sentence-transformers/stsb
language:
- en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: MatterWaveAI/grt-30m
---
# GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)
An **embedding-only Geometric Rotation Transformer (`GRTModel`)** contrastively fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) for semantic textual similarity and dense retrieval using **planar-pair `circular` pooling** (`sentence-transformers`).
Instead of dense linear projections (`nn.Linear`), GRT represents token embeddings in polar coordinates (`phase_angles` + `log_magnitude`) and transforms hidden states through cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations. Because every hidden state is structured as `D / 2 = 128` planar `(x, y)` coordinate pairs, **`circular` pooling** decomposes each pair into its unit direction `(cos θ, sin θ)` and radius `r`, averages the unit directions and radii across non-padding tokens, and recombines them into a `256`-dimensional sentence embedding.
---
## Model Details
| Property | Value |
|---|---|
| **Architecture** | `GRTModel` (`model_type: "grt"`, encoder/backbone without causal LM head) |
| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
| **Pooling strategy** | `circular` (`Pooling(embedding_dimension=256, pooling_mode="circular")` + `Normalize`) |
| **Output dimensionality** | `256` (`128` planar `(x, y)` pairs, L2-normalized) |
| **Total stored parameters** | `29,305,497` (`29.31M`, `lm_head.output_scale` stripped) |
| **Non-embedding backbone parameters** | `9,740,197` (`9.74M` across 12 rotational blocks) |
| **`hidden_size` (`d_model`)** | `256` |
| **`num_hidden_layers` (`n_blocks`)** | `12` |
| **`num_attention_heads` (`n_heads`)** | `4` (`head_dim = 64`) |
| **`intermediate_size` (`d_ff`)** | `1,024` |
| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
| **Vocabulary** | `151,669` (Qwen 3 BPE) |
| **Max sequence length** | `512` (supports up to `2,048` positions) |
---
## Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)
Starting from the pretrained [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) causal checkpoint (`lm_head.output_scale` stripped to form a headless `GRTModel` encoder), the `151,669` lexical `embedding.phase_angles` (`19.41M` parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the **`9,740,196` rotational backbone parameters + `151,669` token `log_magnitude` weights (`9,891,865` trainable parameters)** in `FP32` on an **Intel Arc 140T GPU (`xpu:0`, 32 GB)** using fused oneAPI DPC++ SYCL Givens and Clifford kernels (`~75s` total wall-clock time):
| Stage | Dataset & Budget | Objective & Hyperparameters |
|---|---|---|
| **Parameter Freezing** | `embedding.phase_angles` frozen (`19.41M` params) | `9,891,865` trainable parameters (`12` rotational blocks + `embedding.log_magnitude`); max grad norm `1.0` |
| **Stage 1: AllNLI Contrastive Alignment** | `sentence-transformers/all-nli` (`triplet` split) · `150 steps` (`4,800` triplets, `<1%` of corpus) | In-batch `MultipleNegativesRankingLoss` + explicit hard negatives (`τ = 0.05`), `batch_size = 32`, `max_length = 96`, `PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)` |
| **Stage 2: STS-B Rank & Metric Calibration** | `sentence-transformers/stsb` (`train` split, `5,749` pairs) · `2 epochs` (`358 steps`) | `CoSENT` rank-consistency loss (`τ = 20.0`) + `2.0 × CosineMSE`, `batch_size = 32`, `max_length = 128`, `PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)` |
---
## Evaluation Results & 17-Way Classical Geometric, Symplectic & Toroidal Optimizer Ablation (STS-B)
Evaluated on the **STS Benchmark (`sentence-transformers/stsb`)** validation and test splits using `circular` pooling and cosine similarity across a 17-way controlled optimizer ablation on the identical `508`-step schedule (`150` steps AllNLI + `358` steps STS-B, `seed = 42`):
| Rank (Test) | Optimizer Variant | Coupling LR | Stage 1 NLI Val Spearman (`×100`) | STS-B Val Spearman (`×100`) | STS-B Val Pearson (`×100`) | STS-B Test Spearman (`×100`) | STS-B Test Pearson (`×100`) | Optimizer State Memory (`FP32`) | Step Time (`508` Steps) | Delta vs. `PairAdam` (Val / Test) |
|:---:|---|:---:|---:|---:|---:|---:|---:|---:|---:|---:|
| **1** | **`SymplecticAdam` (Non-Squeezing Floor + Casimir Cavity)** | `1.0×` | `61.98` | `73.60` | `70.87` | **`65.16` (BEST TEST)** | **`65.64` (BEST TEST)** | `80.7 MB` (`2.04×`) | `243s` ROCm | **`+1.81`** / **`+2.74`** |
| **2** | **`HuygensSymplecticAdam` (Huygens Splay Curriculum -> Symplectic Cavity)** | `1.0×` | **`62.82` (#4 S1)** | **`73.68` (TIED BEST VAL)** | `70.90` | **`65.09` (#2 TEST, BEST COMBINED)** | **`65.48` (#2 TEST)** | `80.7 MB` (`2.04×`) | **`236s` ROCm (FASTEST)** | **`+1.89`** / **`+2.67`** |
| **3** | **`PolarNaturalAdam` (Polar Metric + Hessian)** | `1.0×` | `62.31` | `72.62` | `70.20` | **`64.71` (#3 TEST)** | `64.94` (#4 TEST) | `80.7 MB` (`2.04×`) | `4,344s` XPU | `+0.83` / **`+2.29`** |
| **4** | **`GyricAdam` (Thomas–Landé Gyroscopic Projection)** | `1.0×` | `62.21` | `73.04` | `70.23` | **`64.55` (#4 TEST)** | **`65.03` (#3 TEST)** | `80.7 MB` (`2.04×`) | `420s` ROCm | **`+1.25`** / **`+2.13`** |
| **5** | **`HuygensAdam` (Circular Order-Parameter Splay State)** | `1.0×` | **`63.83` (BEST S1)** | `72.95` | `70.35` | **`64.20` (#5 TEST)** | `64.69` | `80.7 MB` (`2.04×`) | `332s` ROCm | **`+1.16`** / **`+1.78`** |
| **6** | **`HessianPairAdam` (`Sophia-Torus`, `1×`)** | `1.0×` | `62.29` | `72.42` | `68.76` | **`63.37`** | `63.69` | `80.7 MB` (`2.04×`) | `2,219s` XPU | `+0.63` / **`+0.95`** |
| **7** | **`BeltramiAdam` (Force-Free Helicity Vortex Cascade)** | `1.0×` | `63.37` (#3 S1) | **`73.63` (#3 VAL)** | `70.68` | **`63.33`** | `63.28` | `80.7 MB` (`2.04×`) | `321s` ROCm | **`+1.84`** / **`+0.91`** |
| **8** | **`HannayAdam` (Adiabatic Holonomy + Polar Metric)** | `1.0×` | `61.15` | `72.57` | `69.90` | **`63.33`** | `63.30` | `80.7 MB` (`2.04×`) | **`239s` ROCm** | `+0.78` / **`+0.91`** |
| **9** | **`RotorAdam` (Cayley Rational Retraction)** | `1.0×` | `62.67` | `73.22` | `70.99` | **`63.29`** | `63.09` | `80.7 MB` (`2.04×`) | `1,926s` XPU | **`+1.43`** / **`+0.87`** |
| **10** | **`ToroidalLion` (Sign-Momentum on Torus)** | `1.0×` | `60.99` | `72.89` | `69.70` | **`63.19`** | `63.41` | **`39.6 MB` (`1.00×`, LOWEST)** | `1,791s` XPU | **`+1.10`** / **`+0.77`** |
| 11 | **`AdamW` (Euclidean Diagonal Baseline)** | `1.0×` | — | `72.89` | — | `63.17` | `63.72` | `79.1 MB` (`2.00×`) | `~1,850s` XPU | `+1.10` / `+0.75` |
| **12** | **`PonceletAdam` (Confocal Elliptic Closure Metric)** | `1.0×` | `62.62` | `72.75` | `69.87` | `63.15` | `63.74` | `80.7 MB` (`2.04×`) | `267s` ROCm | `+0.96` / `+0.73` |
| 13 | **`HessianPairAdam` (`Sophia-Torus`, `3×` Factored)** | `3.0×` | `62.72` | `72.78` | `69.17` | `62.88` | `63.16` | `44.9 MB` (`1.13×`) | `3,698s` XPU | `+0.99` / `+0.46` |
| **14** | **`CyclideAdam` (Dupin Cyclide Conic + Deviatoric Twist)** | `1.0×` | `62.55` | **`73.68` (TIED BEST VAL)** | **`71.79` (BEST VAL)** | `62.87` | **`64.76` (#5 TEST PEARSON)** | `80.7 MB` (`2.04×`) | `380s` ROCm | **`+1.89`** / `+0.45` (`+1.46` Pearson) |
| **15** | **`RindlerAdam` (Rindler Hyperbolic Wedge Horizon)** | `1.0×` | `62.08` | `73.18` | `70.69` | `62.64` | `63.53` | `80.7 MB` (`2.04×`) | `272s` ROCm | **`+1.39`** / `+0.22` |
| 16 | **`PairAdam` (Published Checkpoint, Factored)** | `3.0×` | — | `71.79` | — | `62.42` | `63.30` | `43.4 MB` (`1.10×`) | `~1,820s` XPU | `0.00` / `0.00` |
| 17 | **`ToroidalMuon` (Newton–Schulz Cascade)** | `1.0×` | `63.76` (#2 S1) | **`73.61` (#4 VAL)** | **`71.05` (#2 VAL)** | `61.97` | `63.04` | `80.7 MB` (`2.04×`) | `5,775s` XPU | **`+1.82`** / `-0.45` |
| — | **Pretrained `GRT-30M` (Zero-Shot Base)** | — | `45.51` | `45.51` | `44.14` | `28.49` | `29.81` | `0 MB` | — | — |
### Memory, Speed & Regime Tradeoffs Across GRT Optimizers (`optimization_grt.py`)
For the `9,891,865` trainable parameters (`39.57 MB` in `FP32`: `190,464` rotation angles across `84` cascade tensors, `9,175,040` `Cl(2,0)` coupling parameters, and `526,361` radial scale / norm / `log_magnitude` parameters):
| Optimizer | State Buffers per Param Group | Fine-Tune State Memory (`9.89M` trainable) | Full Pretrain State Memory (`29.31M` all params) | Step Speed (`508` Steps) | Best Operating Regime & Tradeoff Summary |
|---|---|:---:|:---:|:---:|---|
| **`SymplecticAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + symplectic floor & cavity cascade | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `243s` ROCm (`1.02×`) | **Best Test Generalization (`65.16` Test Spearman [BEST] / `65.64` Test Pearson [BEST] / `73.60` Val Spearman, `+2.74` over `PairAdam`).** Enforces Gromov–Liouville symplectic non-squeezing (`c_symp = sqrt(c_base^2 + c_rms^2 * zpe_floor^2 / (u_norm^2 + zpe_floor))`) so vanishing late-stage momentum cannot freeze into sharp training basins, coupled with Helmholtz–Casimir wave-cavity boundary smoothing and Cayley retraction. |
| **`HuygensSymplecticAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + curriculum splay torque & symplectic cavity | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | **`236s` ROCm (`1.00×`, FASTEST)** | **Best Combined Val + Test Spearman (`69.39` Mean Val+Test [BEST] / `73.68` Val Spearman [TIED BEST] / `65.09` Test Spearman [#2] / `65.48` Test Pearson [#2] / `62.82` Stage 1 NLI).** Unifies Stage-1 Huygens circular order parameter (`R_p`) splay-state anti-synchronization with Stage-2 Gromov–Liouville symplectic non-squeezing and Helmholtz–Casimir wave-cavity coupling via a smooth step-and-momentum curriculum gate `gamma_splay(t, m)`. |
| **`PolarNaturalAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + live polar radius `r_p^2` | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `4,344s` XPU | **Best Pure Riemannian Metric Generalization (`64.71` Test Spearman [#2] / `64.94` Test Pearson [#3], `+2.29` over `PairAdam`).** Equalizes arc-length displacement `r_p * d_theta` across high- and low-energy rotation planes via the polar metric `ds^2 = dr^2 + r^2 d_theta^2`. |
| **`GyricAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + Thomas–Landé projection | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `420s` ROCm | **Best Mean (Val + Test) Balance (`73.04` Val / `64.55` Test Spearman [#3] / `65.03` Test Pearson [#2]).** Decomposes cascade velocities into collective orbital (`L`) and stage-differentiating gyric (`S`) components via Landé vector projection (`c_lande = (L^2 - S^2) / J^2`), boosting coherent rotation while retrograde Thomas precession damps incoherent inter-stage fighting. |
| **`HuygensAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + circular order parameter `R_p` | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `332s` ROCm | **Best Stage 1 NLI Zero-Shot Alignment (`63.83` Stage 1 AllNLI [BEST] / `64.20` Test Spearman [#4]).** Tracks the circular synchronization order parameter `R_p = |1/K sum_k exp(i * theta_{k,p})|` and applies a repulsive splay-state restoring torque when stages bunch up, outperforming 5-step Newton–Schulz (`ToroidalMuon`) at a fraction of the compute cost. |
| **`CyclideAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + conic metric & deviatoric projection | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `380s` ROCm | **Best Validation Rank & Pearson Correlation (`73.68` Val Spearman [BEST] / `71.79` Val Pearson [BEST] / `64.76` Test Pearson).** Replaces flat-torus radii with a **Dupin Cyclide** manifold derived from **Brianchon's conic tangent theorem** on the coupled pair ellipse plus a trace-free deviatoric cascade twist. |
| **`BeltramiAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + discrete stage vorticity | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `321s` ROCm | **High-Helicity Cascade Alignment (`63.37` Stage 1 NLI / `73.63` Val Spearman [#2] / `63.33` Test Spearman).** Aligns deviatoric stage updates with discrete stage vorticity (`curl(u) = lambda * u`) to suppress inter-stage shear turbulence. |
| **`HannayAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) + adiabatic scale-angle connection | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | **`239s` ROCm (`1.00×`)** | **Fastest Holonomy-Corrected Polar Optimizer (`72.57` Val / `63.33` Test Spearman in `239s`).** Subtracts the classical Hannay adiabatic holonomy drift `omega_hannay = (r1 * dr2 - r2 * dr1) / (r1^2 + r2^2)` induced by simultaneous scale updates. |
| **`PonceletAdam` & `RindlerAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `267s–272s` ROCm | **Confocal Closure (`PonceletAdam`: `72.75` Val / `63.74` Test Pearson) & Hyperbolic Horizon Contraction (`RindlerAdam`: `73.18` Val / `63.53` Test Pearson).** Interpolates polar/conic metrics by confocal eccentricity (`PonceletAdam`) or contracts the Cayley horizon under proper angular acceleration (`RindlerAdam`). |
| **`RotorAdam`** | Angles: `4×` (`m, v, h, c`) · Non-angles: `2×` (`m, v`) | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `1,926s` XPU | **Smooth Cayley Retraction Baseline (`73.22` Val / `63.29` Test Spearman).** Smooth rational Cayley retraction `Delta_theta = -2 * r_c * arctan(lr * u / (2 * r_c))` on `SO(2)^P` eliminates hard clipping discontinuities. |
| **`ToroidalLion`** | All params: **`1×` (`m` only)** + zero-state Theorem 2.1 saddle kick | **`39.6 MB` (`1.00×`)** | **`117.2 MB` (`1.00×`)** | `1,791s` XPU | **Best Memory-Constrained Optimizer (`72.89` Val / `63.19` Test Spearman).** Cuts optimizer VRAM by **`50%` vs `AdamW` and `70%` vs full-pretrain `HessianPairAdam`** while beating `AdamW` (`63.17`) and `PairAdam` (`62.42`). |
| **`HessianPairAdam` (`1×` / `3×`) & `PairAdam`** | Angles: `2×–4×` · Coupling (`9.18M`): `1.03×` (factored) or `2×` (diag) | `43.4–80.7 MB` | `198.1–388.3 MB` | `1,820–3,698s` XPU | **Foundational Toroidal Baselines (`63.37` Test Spearman for `HessianPairAdam 1×`, `62.88` for `3×` factored, `62.42` for `PairAdam`).** |
| **`ToroidalMuon`** | Angles: `4×` + in-place `5`-step Newton–Schulz `[K, P]` orthogonalization | `80.7 MB` (`2.04×`) | `388.3 MB` (`3.31×`) | `5,775s` XPU | **Stage-Orthogonalized Reference (`63.76` Stage 1 AllNLI / `73.61` STS-B Val Spearman).** |
---
## Usage
### 1. `sentence-transformers` (Recommended)
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")
sentences = [
"Plants convert sunlight into chemical energy through photosynthesis.",
"Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
"The stock market experienced volatility during the afternoon session.",
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities)
```
### 2. Hugging Face `transformers` (`GRTModel` + Circular Pooling)
```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
model_id = "MatterWaveAI/grt-30m-embedding"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()
def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
pairs = token_embeddings.float().unflatten(-1, (-1, 2))
mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
radii = pairs.norm(dim=-1, keepdim=True)
directions = pairs / torch.clamp(radii, min=1e-9)
denom = torch.clamp(mask.sum(dim=1), min=1e-9)
pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)
inputs = tokenizer(
["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
padding=True,
truncation=True,
return_tensors="pt",
)
with torch.no_grad():
outputs = model(**inputs)
embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])
print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))
```
---
## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)
- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
---
## License
Apache 2.0
---
## Citation
```bibtex
@techreport{householter2026grt,
title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
author = {Householter, David},
institution = {MatterWave AI},
year = {2026},
url = {https://matterwave.ai}
}
```